Note that this constraint makes our method more efficient over a brute-force approach, in exchange for loss in recall.
we feel that this trade-off is fair: by retaining the top 1000 candidates, we observed the loss rate of gold standard answers in each of the channels is 14%, 15%, and 17% for phonetic substitution, abbreviation and paraphrase, respectively.