It has been pointed out that evaluating the annotation of a theoretical linguistic notion only intrinsically is problematic because there is no nontheoretical grounding involved (Riezler, 2014).
besides a comparison to the gold standard, we also evaluated the resulting annotation in a larger computational task, the automatic meaning assessment of short answers to reading comprehension questions.