Analyzing the surrogates, we observed that most of the OOV tokens were punctuation or uncommon ideographs, which we expected to, and confirmed to have little effect in the downstream task performance.
we attribute the negligible gains to the nature of the task itself, as it is a news article classification task.