Since this annotation task was a 10-way selection, with multiple selections possible, there were some disagreements.
if categorized broadly into 5W System errors only, MT errors only, and both 5W System and MT errors, then the annotators had a substantial level of agreement (κ=0.75 for error type, on sentences where both annotators indicated an error).