Indeed, it seems unlikely that a model will learn to perform algebraic numerical reasoning based on as few as 50 NLI examples.
a closer look at this dataset provides a potential explanation for this finding.