To measure the efficacy of vector arithmetic in a manner controlled for variances in prior vector similarity, we propose a baseline, defined for each analogy as the best ranking between the word most similar to w 2 and the word most similar to w 3 : For the above example, as dog is the most similar word to dogs, there is no improvement to be made upon baseline.
categories are grouped into a "semantic" and a "syntactic" subset, and results are often reported averaged over each rather than by category.