The main innovation of our architecture lies in disentangling the heads.
instead of concatenating G i to obtain the layer output like the standard Transformer, we employ a binary switch vector s = (α 1 , .