Interestingly, the undirected softmax always benefits the result although it was only used to calculate the penalty in the training step.
the low-dimensional forward and backward representation vectors of DU j are created by concatenating the corresponding final embeddings of tokens w j , w j+1 and the LSTM hidden state of the dependency d ii+1 .