ELMo and BERT improve naive baselines by a large margin, indicating that a notable amount of commonsense knowledge has been acquired via pre-training.
even BERT still falls far behind human performance, indicating the need of further research.