We can see that operating out of their original domains, the automated pipelines, while still outperforming the Stanford pipeline by around 4 percentage points, do not outperform aggregated users.
they do appear to serve well as agents to train participants to perform annotations, as participants annotate to a high level of accuracy.