Of course, a dataset with queries and retrieval scores may serve for more than evaluation; it may provide the examples used by the learning module.
the parameters of the contraction or segmentation scoring function could be discriminatively set to optimize the retrieval of the training set queries.