Some attempts were made to let annotators judge rule correctness directly, that is by asking them to judge the correctness of a given rule (Shinyama et al., 2002;Sekine, 2005).
this simulates the utility of rules in an application and yields high inter-annotator agreement.