Except where noted, we ran MERT three times and report the average score.
we defined a special word <null> whose embedding is the weighted average of the (input) embeddings of all the other words in the vocabulary.