Including an auxiliary ASR task is straightforward with the two-stage model by simply computing the cross-entropy loss with respect to the softmax output of the first stage, and dropping the second stage.
the auxiliary Mt task computes only the second stage, replacing the inputs s src 1:N by states e asr 1:N computed as: instead of computing the secondstage inputs using the first stage, we compute these inputs through a conventional encoder that encodes the reference transcript y transcr 1:N and uses the same embeddings matrix and unidirectional LStM as the first stage decoder.