We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline.
we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset.