In a related study, and aware of the fact that there are often multiple ways to correct individual words in sentence, Rozovskaya and Roth (2010) instead chose to compute agreement at the sentence level.
three raters were asked simply to decide whether they thought 200 sentences were correct or not.