We show below that the more demanding approach of optimising on each subset is superior, especially when extrapolating from small pilot datasets.
if one is not working with computationally efficient algorithms like fastText, one might be tempted to only optimise the hyperparameters once on all the pilot data, and use the hyperparameters optimised on all the pilot data when calculating the error rate on subsets of that pilot data.