Since the output unit biases of the neural network model are independent from the inputs, they correlate strongly (r s = 0.74, p = 0.0) with training corpus word frequencies after being trained.
setting the learned output layer bias vector to a zero-vector is a simple way to reduce the effect of word frequencies on the model's output probability distribution.