Standard word representation models are based on the distributional hypothesis (Harris, 1954) and induce representations from large unlabeled corpora using word co-occurrence statistics (Mikolov et al., 2013;Pennington et al., 2014;Levy and Goldberg, 2014).
as pointed out by recent work (Bojanowski et al., 2017;Vania and Lopez, 2017;Pinter et al., 2017;Chaudhary et al., 2018;Zhao et al., 2018), mapping a finite set of word types into corresponding word representations limits the capacity of these models to learn beyond distributional information, which leads to several fundamental limitations.