Comparing the behavior of the input LSTM and the inference LSTM, we observe interesting shifts of focus.
we see that the inference LSTM tends to see much more concentrated saliency over key parts of the sentence, whereas the input LSTM sees more spread of saliency.