We speculate that current contextualizers are poor at representing named entities like hotels and place names as well as numbers appearing in system and reference texts.
best correlation is still achieved by our word mover metrics combining contextualized representations.