Second, because our model is, in essence, the primary building block of previous models, our results show that some previous papers propose needless complexity, and that gains from these previous complex neural architectures are quite modest.
the state of the art is achieved by careful tuning of simple and wellengineered models, not necessarily by adding more complexity to the model, echoing the sentiments of Lipton and Steinhardt (2018).