Indeed, we often see probing models rivaling or exceeding the performance of (often carefully tuned and task-specific) state-of-theart models.
the linear probing model surpasses the published state of the art for grammatical error detection and preposition supersense identification (both role and function).