We adapt our parsing algorithm accordingly to produce k most probable DTs for each text, and measure the oracle accuracy based on the F-scores of the Relation metric which gives aggregated evaluation on structure and relation labels (see Table 2).
the oracle accuracy O−score for k-best discourse parsing is measured as follows: where N is the total number of texts (sentences or documents) evaluated, g i is the gold DT annotation for text i, h j i is the j th parse hypothesis generated by the parser for text i, and F-score r (g i , h j i ) is the F-score accuracy of hypothesis h j i on the Relation metric, which essentially measures how similar h j i is to g i in terms of its structure and labels.