For quality, we compare the search result sets for the autocorrected queries with those of the reference and get an estimate of how similar each autocorrected query behaves to the expected behavior given by its gold correction.
we extract the categories from all items which are returned as part of the search API calls (see Table 7).