BERT is intended to be pre-trained in the described manner and then fine-tuned to a specific task.
as our goal is not to solve a specific task, but rather to determine the informational content of a model, we employ the featurebased approach without fine-tuning presented in the original work.