We present (Section 3) the Pivot Based Language Model (PBLM) -a domain adaptation model that (a) is aware of the structure of its input text; and (b) outputs a representation vector for every input word.
the model is a sequential NN (LSTM) that operates very similarly to LSTM language models (LSTM-LMs).