Trained annotators are more correlated with one another (ρ = 0.90, α = 0.67, κ = 0.34) than the crowdsource annotators (ρ = 0.75, α = 0.52, κ = 0.09).
the model achieves better results using annotator rather than author ratings, even when controlling for rating variance as a confounding factor.