The second group (which, what and how) are at 79-82% percentage agreement between crowd and gold standard annotation, which is likely due to their more ambiguous answer surface realization possibilities, e.g., a what-question can ask for an activity ('What did Peter do?')
with respect to the observed differences between the annotation quality of the answers to different question form subtypes, our hypothesis is that since certain examples, such as answers to why-questions, exhibit a much greater variation in terms of their linguistic material, this leads to less consistent results in the annotation, especially for the crowd.