In each iteration, there are two cases: • Case 1: If the perceived relevance of the snippet of document d 1 is larger than that of d 2 , the sentence ranking function should not be modified; • Case 2: If the perceived relevance of the snippet about document d 2 is larger, then we should modify the sentence ranking function so that the snippet of d 1 becomes more relevant and that of d 2 less relevant; In Case 2, the modification of sentence ranking function f is achieved by adding new sentence preference pairs into the training set.
for document d 1 , we should rank higher the sentences that are more relevant than the snippet of d 2 : In our implementation, d ′ is a subset of d 1 composed by the sentences that contain at least one non-stop query term.