To make the evaluation realistic, we used a different method to create the test set: we paired each query word with each of the 771 remaining words to form the test set.
in each fold, the training set had an equal number of positive and negative pairs, while in the test set, negative pairs outnumbered the positive pairs.