In the realm of computer vision, PreAct-ResNet (He et al., 2016b) demonstrated that it is helpful to apply batch normalization to x, instead of F(x, W).
the batch normalization acts on the output of the previous block.