Although ELMo, BERT, and GPT-2 have 2, 12, and 12 hidden layers respectively, we also include the input layer of each contextualizing model as its 0 th layer.
the higher the layer, the more contextspecific the contextualized representations.