In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset.
the context vector is highly contextualized by the current sequence.