Attentive Readers To enable the model to gather information from a longer context, we aug-ment the supervised LSTM model with the attention mechanism (Bahdanau et al., 2014), so that the representation at the blank is used as a query to find the relevant context in the document and a blank-specific representation of the document is used to score each candidate answer.
we adapt the Stanford Attentive Reader (Chen et al., 2016) and the positionaware attention model (Zhang et al., 2017) to the cloze test problem.