Various statistical methods have also been proposed to filter out the mistakes or (spamming) random responses of the † Corresponding author crowdsource workers (Liu et al., 2012;Hovy et al., 2013;Nguyen et al., 2017).
the way to filter out the mistakes or the random responses through statistical means is difficult to utilize for a fundamentally subjective annotation task.