Concerns have been expressed as to the quality of crowdsourced data, which some assess as part of a trade-off for speed and economy (Snow et al., 2008;Madnani et al., 2011;Ball, 2014), with others describing methods to filter out errors (Gadiraju et al., 2015;Schmidt et al., 2015), or indeed encouraging researchers to 'embrace' error (Jamison and Gurevych, 2015;Krishna et al., 2016).
as the old adage goes, you only get what you pay for.