These benchmarks help to provide a uniform evaluation of new modeling developments.
recent work shows a problem with this standard evaluation paradigm based on i.i.d. test sets: datasets often have systematic gaps (such as those due to various kinds of annotator bias) that (unintentionally) allow simple decision rules to perform well on test data (Chen et al., 2016; Gururangan et al., 2018; Geva et al., 2019). 