A structured regularization is applied to the embeddings to ensure they are tuned in an incremental way.
the tuned word embeddings become better word representations by absorbing semantic information from supervision without "forgetting."