We linearly project the last representation h t using F ∈ R d model ×d model for querying W .
we extend Transformer (Vaswani et al., 2017) for supporting insertion operations, where the generation order is directly captured as relative positions through self-attention inspired by Shaw et al.