For word w, we estimate the mean vector µ w with the average over n-gram vectors and its dictionary-level vector.
where z g is a vector associated with an n-gram g, v w is the dictionary representation of word w, and N G w is a set of n-grams of word w. Examples of 3,4-grams for a word "beautiful", including the beginning-of-word character ' ' and end-of-word character ' ', are: • 3-grams: be, bea, eau, aut, uti, tif, ful, ul • 4-grams: bea, beau .., iful ,ful This structure is similar to that of FASTTEXT (Bojanowski et al., 2016); however, we note that FASTTEXT uses single-prototype deterministic embeddings as well as a training approach that maximizes the negative log-likelihood, whereas we use a multi-prototype probabilistic embedding and for training we maximize the similarity between the words' probability densities, as described in Sections 3.2 and 3.3 Figure 1a depicts the subword structure for the mean vector.