The input is the embeddings of the current word and its left context The output is given by: In fact, absolute distance values are not required, it is sufficient to preserve their order.
if   Predict next word Gated-weighted attention Figure 3: The prediction of the next word in the PRPN language model.