Partial-input baselines are valuable sanity checks for datasets, but as we illustrate, their implications should be understood carefully.
for example, partial-input baselines are used to identify the "hard" examples in SNLI (Gururangan et al., 2018), verify that SQuAD is well constructed (Kaushik and Lipton, 2018), and that SWAG is challenging (Zellers et al., 2018).