While it may be surprising that GloVe initialization is harmful, it is well known that pretrained word embeddings do not necessarily capture syntactic relationships (as evident in the poor performance of k-means clustering).
given random word y ∈ V , we set x ∈ V 2H to be an ordered list of H left and H right words of y.