We could not consider the existing negative keyword lists from the campaign as a gold standard because they were incomplete, for only a few topics; they also contained intentional misspellings (something that this system does not consider); they contained negative keywords that were related to user intention instead of meaning (such as car rentals instead of purchases); and they contained a few emotional negative keywords indicating that the audience had negative sentiments towards the focus of the ad.
we used a number of ad hoc tests to determine the effectiveness of our system.