As such, we hope that these evaluation datasets can help guide multilingual language modeling research across a wide spectrum of languages.
our goal now is to verify that type/token ratio and not absolute corpus size is the deciding factor when unraveling the limitations of standard LM architectures across different languages.