As Turney observes, this approach seems to have the effect of underestimating model quality relative to the interannotator agreement figure, which is calculated as average intersubject correlation.
a further advantage of the probabilistic framework is that it is often straightforward to extend existing models to account for additional structure in the data, or to tie together parameters for shared statistical strength, while maintaining guarantees of well-normalized behavior thanks to the laws of probability.