These annotations are the concatenation of the matching forward and backward hidden layer states obtained respectively by the forward and the backward RNN comprising the bidirectional RNN.
they contain the summaries of the dialogue turn contexts respectively preceding and following a considered word.