W rh acts like a key phrase detector that learns to capture the pattern of generation tokens or the relationship between multiple tokens.
the new input x t consists of information of the original input token w t , the DA representation d t , and the hidden context h t 1 .