We are curious how big the effect of the pre-trained model is.
instead of starting with the BERT-BASE-UNCASED weights, we initialize BISON with random weights drawn from a normal distribution with mean 0.0 and standard deviation of 0.02.