These input embeddings can either be learned by the model or be fixed to a pretrained embedding.
the last hidden state computed by this LStM is then transformed linearly to yield an m-dimensional definition embedding h. the encoder whose parameters are θ = {E, Ω, W, b} computes this embedding h as the subsequent decoder can be seen as a conditional language model trained by maximum likelihood to regenerate definition D w given definition embedding h = f θ (D w ).