One way to capture the contextual information of a word sequence is to concatenate neighboring features as input features for a deep neural network.
the number of parameters rapidly increases according to the input dimension.