For each time step t, a neural network builds a representation h t from the history that is used to predict the upcoming character.
this technique was previously considered in the context of automatic speech recognition (Wang et al., 2017) or to copy tokens from the input for code generation (Ling et al., 2016).