Other recent work has shown that even state-of-the-art neural models struggle with numerical operations and quantitative reasoning when trained in an end-to-end manner (Saxton et al., 2019;Ravichander et al., 2019).
even BERT (Devlin et al., 2019) is not very good at doing simple calculations.