However, we have demonstrated some limitations of statistical significance testing for NLP.
while statistical significance is usually a minimum necessary condition to demonstrate that a performance difference is real, it's also important to consider the relationship between test set performance and the actual goals of the systems being tested, especially if the system will eventually be used on data from a different domain than the test set used for evaluation.