The total losses are minimized by an Adam optimizer (Kingma and Ba, 2015) with 0.0001 learning rate.
the size of lowlevel clusters and high-level clusters are 3400 and 340, respectively, which are significantly smaller than the size of raw words (34000), and these clusters are also represented by 300-dimensional vectors.