The former primarily generate general purpose and domain independent embeddings of word sequences (Socher et al., 2011;Kiros et al., 2015;Arora et al., 2017); many unsupervised training research efforts have focused on either training an auto-encoder to learn the latent structure of a sentence (Socher et al., 2013), a paragraph, or document (Li et al., 2015); or generalizing Word2Vec models to predict words in a paragraph (Le and Mikolov, 2014;Chen, 2017) or in neighboring sentences (Kiros et al., 2015).
some important information could be lost in the resulting document representation without considering the word order.