Similar to WSD, the biLM representations are competitive with carefully tuned, task specific biLSTMs (Ling et al., 2015;Ma and Hovy, 2016).
unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).