Throughout the entire experiments, we mainly used huggingface’s seven pre-trained transformers2 , implemented with Pytorch.
since the original implemented models do not return the output vectors of the internal attention heads, we developed the wrapper class that enables extracting the output vectors from the created pre-trained model objects.