We did not directly measure the difference in attention cost, but it cannot be greater than the difference in total cost for the smallest number of layers.
we can infer from Figure 3 that processing 5 layers in a character model incurs roughly the same time cost as attention.