This seems to be unreasonable in practice, especially for a long text segment such as a paragraph or a document, where the interaction between the first word and the last word is often negligible.
we address this limitation by proposing a local mixture of words, which tends to capture the semantic relations between neighboring words and undermine the long-range word dependencies.