SWEMs bear close resemblance to Deep Averaging Network (DAN) (Iyyer et al., 2015) or fast-Text (Joulin et al., 2016), where they show that average pooling achieves promising results on certain NLP tasks.
there exist several key differences that make our work unique.