One major limitation is that attention at each time step is not directly dependent of each other.
in machine translation, the next word to attend to highly depends on previous steps, neighboring words are more likely to be selected in next time step.