5, we can find that contextindependent word representations can not contribute much to current models.
when the models are equipped with BERT, we are excited to observe that the performances of all types of architectures are improved by a large margin.