While these results are rather encouraging, they were not optimal.
metrics used to measure the performance of automatic systems against the human output were borrowed from the clustering literature (Hotho et al., 2003) and they are defined as follows: where C is the set of clusters to be evaluated and L is the set of clusters produced by the human.