Although complex compositional functions are avoided in these models, additional modules, such as attention layers, are employed on top of the word embedding layer.
the specific role that the word embedding plays in these models is not emphasized (or explicit), which distracts from understanding how important the word embeddings alone are to the observed superior performance.