While these results are encouraging and hint at the ability of very large pretrained language models to memorize factual knowledge, the large GPTv2 model has not been made public and the publicly available small version achieves less than 1% on Natural Questions (5.3 times worse than the large model).
we decided to not include GPTv2 in our study.