The first assumption we make to minimize the number of human judgments, is that once we have a judgment for a source-candidate substring pair, that same judgment can be used across all candidates for this source sentence.
we build a database for each source sentence, which consists of <source substring,target substring,judgment> entries.