Network structures: We used both feed-forward networks with tanh non-linearities and LSTM (Hochreiter and Schmidhuber, 1997) networks.
for the main experiments, we evaluate on two corpora: the Penn Treebank (PTB) data set prepared by Mikolov et al.