Also, we were using a 7-item Likert scale, whereas the WSsim/USim team used a 5-item Likert scale, and, intuitively, increased granularity ought to increase the risk of interannotator disagreement.
wSsim/Usim were purposefully annotated by non-experts, whereas all our annotators are linguists (with high non-native proficiency in English) and had been working with PDEV before.