In recent years, Deep Learning methods have become very popular in Natural Language Processing (NLP), e.g., they reach high performances by relying on very simple input representations (for example, in (Kim, 2014;Goldberg, 2016;Kim et al., 2016)).
transformer-based architectures, e.g., BERt (Devlin et al., 2019), provide representations of their inputs as a result of a pre-training stage.