At each recurrent step, information exchange is conducted between consecutive words in the sentence, and between the sentence-level state and each word.
each word receives information from its predecessor and successor simultaneously.