We have worked with a limited number of eight journalists, as it is cumbersome and expensive to obtain domain expert annotations, but the crowd-sourcing platforms are generally low-cost and employ a very high number of annotators for their tasks (in our case eighty).
the non-experts have annotated generally more articles and also more articles in common with each other -the latter could make their agreement scores more robust.