Typically, highway layers conduct nonlinear transformation as: where σ is the sigmoid function; ⊙ is the elementwise product operation; g is a nonlinear activation function; t is called transform gate and (1 − t) is called carry gate.
when taking each token as the current word, we get the representation D from all tokens calculated by GCNs.