Ranking Transformer models are pretrained on Reddit data (Mazaré et al., 2018) and fine-tuned.
we adapt the BERT contextual language model (Devlin et al., 2018) to the task of dialogue in two ways: as a bi-ranker, which is fast and practical as a retrieval model, and as a cross-ranker which is slower at inference time but allows more feature cross-correlation between context and response.