Although the RNN is able to capture long-span context information to some extent, the local context may very easily dominate the hidden state.
this additional soft headword embedding allows the model to access long-span context information in a different channel.