First, for all encoders, the two-way attentions provide consistent enhancement to the baselines with different types of knowledge.
although the baseline model is well-performed when BERT (or ZEN) serves as the encoder, the attention module is still able to further improve its performance with the knowledge produced by the toolkits even though the toolkits have worse-than-baseline results for the joint task