Moreover, we see that this effect is not just driven by sensitivity to the specific vt bigram.
simply swapping "to" with "that" (or vice-versa) in a naturally-occurring context leads to a small  shift in the distribution of the model's predictions away from the target D vt distribution, but not to the same degree as replacing a "that"-taking verb with a "to"-taking verb in a naturally-occurring "that" context (or vice-versa).