Their removal led to a correlation reduction of 10.2% (rounded) and 6.2% (continuous) relative to using all features.
spam and malicious workers were identified during data collection using a filtering algorithm that compared annotations with those completed by "potentially good annotators" (P GA).