With bidirectional RNN, the dependencies in a sentence from both directions can be captured, and with L-STM architecture, long-range dependencies can be well modeled.
as shown in Table 2, compared to standard pre-training, the influence of heterogenous data is more evident.