In addition, Table  3 shows that increasing the distributions from the final layer (i 3 = 20) degraded the score from the language model with i 3 = 15 (the top row of Table 3).
to obtain a superior language model, we should not increase the number of distributions from the final layer; we should instead use the middle layers, as with our proposed DOC.