The Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) or the more recently introduced gated recurrent unit (GRU) (Chung et al., 2014) use gates to control the information flow from previous words.
lSTM and GRU are better at capturing longterm dependencies than simple RNNs, and are often chosen in practice.