Indeed, in most experimental NLP papers, several algorithms are compared on a number of datasets where the performance of each algorithm is reported together with per-dataset statistical significance figures.
with the growing number of evaluation datasets, it becomes more challenging to draw comprehensive conclusions from such comparisons.