If one were to report unweighted precision and recall without using any bins-as is the norm-this important qualification would have been harder to discover.
we show that crowdsourcing yields judgments that are as good but without the cost.