Finally, the expert gates G use token-level NLL loss in (2), which translates to the negative loglikelihood of using the correct language expert for each token in a sample.
the objectives that C, D and G minimize are, respectively: where h w in (5) is the BiLSTM hidden representation in F p as shown in Figure 2.