3 • F is independent if the collective annotation of any given item j only depends on the individual annotations of j.
by and large, the emphasis so far has been on issues such as experiment design, data quality, and costs, with little attention being paid to the aggregation methods used, which are typically limited to some form of majority vote (or taking averages if the categories are numeric).