We used a common task in NLP, text classification, on a relatively recent but widely used data source, Twitter streams, to show that blindly following the same evaluation method for tasks of similar nature could lead to invalid conclusions.
we investigated the most common evaluation method for machine learning applications, standard 10-fold cross-validation, and compared it with other validation methods that take the statistical dependence in the data into account, including time-split, bordersplit, neighbor-split, and a combination of time-and border-split validation.