As the empirical guessability increases, so does the median model probability assigned to the correct answer.
in our applications, we are less interested in only the 1-best prediction; we'd like to know whether users can understand the novel vocabulary, so we'd prefer to allow WordNet synonyms to also be counted as correct.