We only preserve word types that appear more than 100 times and replace all others with a special symbol, resulting in a vocabulary of size around 188k.
another novelty of our work is considering a lowrank approximation of a linear regressor that predicts the context from words.