First, for each pair of entities (h, t) that appear in the test set, all the corresponding mentions M h,t in the test set are collected and a prediction is performed with: The predicted relationship can either be a valid relationship or NA -a marker that means that there is no relation between h and t (NA is added to R during training and is treated like other relationships).
ifr h,t is a relationship, a composite score is defined: only the top scoring non-NA predictions are re-scored.