We use the same evaluation metrics as used for the lexical substitution task at Semeval-2007.
we measure the precision and the recall for four subtasks: best normal, which measures the precision and recall obtained when the first synonym provided by the system is selected; best mode, which is similar to best normal, but it gives credit only if the first synonym returned by the system matches the synonym in the gold standard data set that was most frequently selected by the annotators; out of ten (oot) normal, which is similar to best normal, but it measures the precision and recall for the top ten synonyms suggested by the system; and out of ten (oot) mode, which is similar to best mode, but it again considers the top ten synonyms returned by the system rather than just one.