This is surprising as the embedding directly provides a number's value, thus, the synthetic tasks should be easy to solve.
we had difficulty training models for large ranges, even when using numerous architecture variants (e.g., tiny networks with 10 hidden units and tanh activations) and hyperparameters.