PaLM provides an intuitive way to inject structural inductive bias into the language model-by supervising the attention distribution.
their contribution is orthogonal to ours and not head-to-head comparable to the models in the  punctuation removal.