In fact, our system corrects the relation labels of the above 6 sentences before training the relation extractor.
the sentential precision and recall is computed on a randomly sampled set of sentences from S e ∪ S F , in which each sentence is manually labeled whether it expresses any relation in R. Figure 3 shows the precision/recall curves for MULtIR with and without pseudo-relevance feedback computed on the test dataset of 1000 sentence used by Hoffmann et al.