3, where L is the number of hidden layers).
a word is predicted using both its left and right context jointly, akin to the context2vec architecture (Melamud et al., 2016) but differently from, e.g., the BiLSTM architecture used for ELMo (Peters et al., 2018).