As will become clear in what follows, the Encoder of the QBERT architecture is able to compute the hidden representation of a word w t in a sequence w 1:n as a function of the weights and of the whole sequence except w t itself, i.e.
with respect to our own comparison systems, QBERT performs better than ELMo, Flair and SBERT in this setting as well.