As a first step to examine the quality of the human labels, we measure the inter-annotator agreement (ITA) within each collection.
we calculate the agreement for the expert dataset, the Figure Eight dataset and the MTurk dataset separately.