We assume that a hidden stateh i ∈ R 2d is computed for each x i by using a bi-directional RNN with LSTM units (Graves and Schmidhuber, 2005).
,h i is the concatenation of computed by a pair of forward and backward RNNs.