Compared to the second class, we learn from free-form text through MLMs rather than triples, which fosters generalization on other downstream tasks.
given a corpus of raw text and a KG, two KG-guided self-supervision tasks are formulated to inject structured knowledge into MLMs.