We evaluate how well CWR features perform the pretraining task-bidirectional language modeling.
we take the pretrained representations for each layer and relearn the language model softmax classifiers used to predict the next and previous token.