Collecting human judgements is the method of choice for evaluating sentence compression models.
human evaluations tend to be expensive and cannot be repeated frequently; furthermore, comparisons across different studies can be difficult, particularly if subjects employ different scales, or are given different instructions.