To further improve performance, we designed PYTHIA's encoder to take an additional input stream of word embeddings, as it is difficult to model the word-level context using only character-level information.
we generated a list of the 100k most frequent words appearing in PHI-ML, and using a separate lookup table we concatenated at each time-step the embedding of each character with the embedding of the word it belongs to.