The parameters in GPT are all the same for all wordpiece embeddings in the vocabulary.
we choose it as a part of our evaluation corpora because it requires the model to capture rich linguistic phenomena.