Layer 1 and 2 alignments are based on anchors from the first and second LSTM layer output respectively.
we emphasize that by constraining w s→t to be orthogonal, we also preserve relations between e i,c andê i,c that represent the contextual information.