To make sure that the IAA of good squared and good annotations is not the result of a different sample size, we randomly sampled 10%, 20%, ..., and 90% of all the annotations.
we adopted random sampling as a baseline method to compare with our proposed method of the twostep reason selection, and checked the effect of the sample size on IAA.