For each model and training size, we averaged the perplexities from 5 trials of each competition track.
couldn't a system be rewarded simply by being compared against a bad system more frequently than its competitors?