We first introduce a bi-directional LSTM (BiL-STM) as the primary-capsule layer to incorporate forward and backward context information of the input sentence: Here we produce the sentence-level encoding h t of word x t by concatenating the forward − → h t and backward output vector ← − h t .
the output of BiLSTM encoder are a sequence of vectors H corresponding to the input sequence.