However, it is hard to guarantee that every turker performs the survey carefully, given the difficult and long passages of high school problems.
to obtain the ceiling human performance on RACE, we manually labeled the proportion of valid questions.