We assess the system's accuracy on the newly labeled data by comparing the system's labels with the human's new labels.
accuracy for +effect and -effect is calculated such as: accuracy +ef f ect = # annotated +effect # top 5% +effect data accuracy −ef f ect = # annotated -effect # top 5% -effect data the accuracy means that out of the top 5% of the +effect (-effect) data as scored by the system, what percentage are correct as judged by a human annotator.