Transformer Our transformer (Vaswani et al., 2017) model uses 300 dimensional embeddings and hidden states, 2 layers and 2 attention heads with no dropout.
we want to learn a conditional probability distribution P θ (y|x) where y is a reasonable response given the conversation history x.