We conduct an experiment using a BERT model without the finetuning process and we find the final dev label accuracy is close to the result from a random guess.
the fine-tuning process rather than sentence embeddings plays an important role in this task.