This decision step is accomplished by a simple feed-forward network (Figure 3).
replacing GrU with Long-Short Term Memory (LSTM) cells (Hochreiter and Schmidhuber, 1997) gives similar performance but the training procedure is computationally more expensive.