Multitask pretraining or intermediate task training offers modest further gains.
we see several worrying trends: • The margins between substantially different pretraining tasks can be extremely small in this transfer learning regimen and many pretraining tasks struggle to outperform trivial baselines.