Just as direct training data are scarce, ground truth for relation extraction is scarce as well.
prior work mainly considers two types of evaluation methods: (1) randomly sample a small portion of predictions (e.g., top-k) and manually evaluate precision/recall; and (2) use a held-out portion of seed facts (usually Freebase) as a kind of "distant" ground truth.