This model is then used to predict the quality of the comments in the 1/10th sample and compared to the true quality for those comments (using Pearson correlation in this case).
many of our scores for change in forecaster accuracy (benefit) are based simply on one change and thus quite unreliable.