The increases in loss are recorded in Table 5, which shows that [Next,Prev] patterns are most important, followed by [First] and [Period], while [Delim] isn’t very useful
we find that RoBERTa-AG is very robust and does not need much tuning.