Inter Annotator Agreement is the only way to judge the reliability of annotated data in absence of an end application.
before we can venture into analysis of the experimental data, we need to formalize the notion of IAA for flat and nested queries.