The group recurrent layer in Equation 4 and 5 can be re-formulated as From the reformulation, we can see group recurrent layer is equivalent to standard recurrent layer with block-diagonal sparse weight matrix.
training such big models usually takes days or even weeks of time even if using tens of GPU cards.