To assess the contribution of different components of PretRand, we performed an ablation study.
we successively ablated the main components of PretRand, namely, the learnable vectors (learnVect), the longer training for random units (random++) and the normalisation ( 2 -norm).