We define the new attention score and context vector asα andc, which are computed as: From the definition above, it can be inferred that when the temperature increases, the distribution of the attention score α is smoother, meaning that softer attention is required, and when the temperature is low, the distribution is sparser, meaning that harder attention is required.
the model can tune the softness of the attention distribution self-adaptively based on the current output for the decoder and the history of attention, and learns when to attend to only corresponding words and when to attend to more relevant words for further syntactic and semantic information.