Note that this context gate is crucial to avoid retraining on all data or repeatedly duplicating a large base model every time.
the lateral connections transfer the hidden states t−1 i for each word i from M t−1 to the newly expanded component M t .