Note that, Equation 8 indicates that the representation y l depends on both s + l and y l−1 .
although Equation 7 states that the gate activation at layer l is computed by s + l , the gate activation is also affected by y l−1 , which embeds the information from the layers below l. Intuitively, the AGT networks are encouraged to consider new words/phrases from the input text at higher layers.