How should we evaluate rankings?
we follow the suggestion in Ravichandran, Pantel, and Hovy (2005) of reporting the percentage of agreements in the top-S. we compare the top-S pairs based on a sample with the top-S pairs based on the entire data set.