Pattern matching alone may generate inappropriate pairs, so we then assess the legitimacy of each collected slot-filler.
human evaluators were then asked to score each pair of an original sentence and a paraphrased sentence with the following two 5-point scale grades proposed by Callison-Burch (2008): Grammaticality: whether the paraphrased sentence is grammatical (1: horrible, 5: perfect) Meaning: whether the meaning of the original sentence is properly retained by the paraphrased sentence (1: totally different, 5: equivalent) To make results more consistent and reduce the human labor, evaluators were asked to rate at the same time several paraphrases for the same source phrase.