12 All these efforts face the problem of how to aggregate the information provided by a group of volunteers into a collective annotation.
by and large, the emphasis so far has been on issues such as experiment design, data quality, and costs, with little attention being paid to the aggregation methods used, which are typically limited to some form of majority vote (or taking averages if the categories are numeric).