Representing history To represent x <t , we use an LSTM encoder to read the sequence of characters, where each character type σ ∈ Σ has a learned vector embedding v σ .
the history representation at time t is This corresponds to the standard history representation for a character-level language model, although in general, we assume that our modelled data is not delimited by whitespace.