We implement BISON based on the BERT Pytorch code 1 and initialize with the pre-trained BERT model BERT-BASE-UNCASED (Devlin et al., 2018).
we employ the same model architecture and tokenisation as (Devlin et al., 2018) resulting in a model with about 110M parameters.