Including an auxiliary ASR task is straightforward with the two-stage model by simply computing the cross-entropy loss with respect to the softmax output of the first stage, and dropping the second stage.
as a remedy, we propose the use of endto-end trainable models with two attention mechanisms, the first establishing source speech to source text alignments, the second modeling source to target text alignment.