Previous neural network models (Pei et al., 2015;Pei et al., 2014;Zheng et al., 2013) normally set context window around a word and extract atomic features within the window to represent the contextual information.
context window limits their ability in detecting long-distance information.