This is likely to bias representations by speakers from different countries talking about specific issues or places (as in Figure 2), and gives the model comparatively little information to work with as there is no explicit syntactic information available.
of the lack of explicit syntactic information, it is unsurprising that the results (LM-Raw in Table 1) only marginally outperform the random baseline.