The improvements are robust and significant on both tasks, both metrics, and on all depths.
the transformer is convolutional on all length k of n-grams; the same parameters are used model the interactions between n-grams at each length, to reduce the parameter size.