In our exploration of architectures for the sentence encoding model, we also tried using a selfattention layer following the intuition that not all words are equally important for the meaning of a sentence.
we later realized that the cross lingual sentence similarity objective is at odds with what we want the attention layer to learn.