The neural counterpart to alignment, attention (Bahdanau et al., 2015), which is a key part of our approach, was originally proposed and has been predominantly used in conjunction with LSTMs (Rocktäschel et al., 2016;Wang and Jiang, 2016) and to a lesser extent with CNNs (Yin et al., 2016).
our use of attention is purely based on word embeddings and our method essentially consists of feed-forward networks that operate largely independently of word order.