We use the notion of additive token embedding as in (Botha and Blunsom, 2014) to utilize the available information about the token, e.g., its word form, lemma, part-of-speech (POS) tag, and morphological features.
the token embedding is computed as where e i 's are one-hot encoding vectors for the ith word, and E's are parameters to be learned that store the continuous embeddings for corresponding feature.