Formally, suppose that BM @N is the top N questions retrieved by the benchmark model, and RS@N is the top N questions retrieved by a method we want to test.
besides the automatically harvested knowledge bases, other types of knowledge such as a traditional Db (Merzbacher, 2002) or a linked open data (Marie et al., 2013) are manually generated and often do not cover new entities, like new idols possibly attracting click-through.