As a second way of evaluating the quality of our automatic labeling, we use these silver labels in a semi-oracle trial where we apply the actual simplification operations as given in the annotated corpus.
we simply take the automatic labels as true and use the A  D M R RM RW RWM  C  O   A  240 0  0 0  0  2  0  0  19  D  15 333 8 4  5  1  1  4  0  M  0  1 22 0  0  0  0  1  0  R  0  33 0 28 6  0  0  4  0  RM  0  8  0 0  4  0  0  0  0  RW  3  31 4 7  2  4  0  6  0  RWM 0  6  0 0  0  0  0  0  0  C  0  24 98 1  1  1  0  1807  0  O 105 0 0 0 0 9 0 0 1998 Table 4: Confusion matrix of true (rows) and automatically annotated (columns) operations on the manually annotated data.