For each of the proposed rules from all four systems, we collect human judgements on Amazon Mechanical Turk, using the same annotation interface as before.
we ask 7 workers to view each pair and indicate which of the two phrases is simpler, or to indicate that there is no difference.