In addition, Da and Kusai (2019) proves that pre-trained language models have the ability to encode some commonsense knowledge in the embedding space through the attribute classification evaluation.
from figure 5, we can see that the layout of question, the correct answer 'garage' and the wrong answer 'utility room' are 'a person' with 'a truck', 'cars', and 'chairs' and 'old televisions', respectively.