The first hidden layers have recurrent connections from the past words while second one's direction of recurrent of connections is flipped, passing activation backwards in the texts.
in the LSTM layer, we can get the forward hidden state h i,j from the forward LSTM network and the backward hidden state h i,j from the backward LSTM network.