Namely, our method achieves an accuracy of 60.3%, nearly 3% of accuracy above the previous best result.
the concept of self-attention is further expanded within BERt by the idea of so called multi-head outputs that are incorporated in each layer.