By including the same target word in each element of the pair, the dataset is constructed in such a way that contextinsensitive word embeddings would not perform better than the random baseline.
the dataset is an ideal evaluation set for assessing the quality of the semantic information encoded in contextualized embeddings.