To enforce monotonicity, hard attention with conditionally independent alignment decisions is not enough: The model needs to know the previous alignment position when determining the current alignment position.
we allow the transition distribution to condition on previous one alignment p(a i | a i−1 , y <i , x) and it becomes a 1 st -order neuralized HMM.