BERT (Devlin et al., 2018) fine-tuned on QQP achieves over 90% accuracy on QQP, but only 33% accuracy on PAWS data in the same domain.
in our experiments, we show the limitations of this modeling choice on PAWS pairs.