To evaluate the performance of our approach, we employ the measures in (Luo, 2005), which are computed as follows, where R ' is the predicted partition and R is the ground-truth partition; π(A, B) is a similarity measure between set A and B, which is Jaccard coefficient in this paper; and g(.)
for a given query q, we collect its reformulations with additional words from query log as query subtopic candidates, e.g., we collect {'jaguar sports car', 'jaguar XJ sport', 'jaguar diet', …} for query 'jaguar'.