Hence, their training was deterministic and the number of configurations a model could have was rather small -decisions about model design were usually limited to feature selection and the selection of one of a few loss functions.
when one model performed better than another on unseen data it was safe to argue that the winning model was generally better, especially when the results were statistically significant (Dror et al., 2018), and when the effect of multiple hypothesis testing was taken into account in cases of evaluation with multiple datasets (Dror et al., 2017).