In literature, there is a large bundle of studies for injecting some level of word order information into word embeddings (Qiu et al., 2014;Johnson and Zhang, 2015;Lai et al., 2015;Ling et al., 2015;Trask et al., 2015) or into general word-space models (Sahlgren et al., 2008;De Vine and Bruza, 2010;Basile et al., 2011).
we refer to the work of (Trask et al., 2015) that extends the CBOW model proposing to split the context into two partitions, the first containing the sum of all the context word embeddings before the target word and the other containing the sum of all the word embeddings after.