When using more powerful contextualized ELMo and BERT word embeddings (the third and fourth section of Table 3 respectively), we observe similar trends but with larger performance gains overall.
compared to its equivalent word expert model Word-Specific Classifier BiLST M +Gloss , the Single Classifier BiLST M +Gloss achieved clear performance gains of 2.1% and 2.0% in F1score on the combined test set, when using ELMo embeddings and BERT embeddings respectively.