To address this issue and directly improve the quality of character alignment, we add a residual connection from the encoder embedding layer to the decoder projection layer.
letting α be the predicted attention weights, we compute where E x (i) is the encoder character embedding at position i, and c is the weighted character embedding.ĥ is subsequently used to predict the next character.