During training, the model learns to embed "one-hot" words into a lower 500d space by applying a linear transformation.
the embedding is learned only from the limited and possibly noisy text in the caption data.