So the short answer is that this second annotation was not worth the effort, and we hope that this can be of use for future work on the development of such evaluation data.
the comparison of the two data sets gives us useful insights on the task itself, and helps us understand the gaps between the systems.