All supervised objectives are optimized jointly with the encoder being shared, resulting in the following multi-task learning objective: Adversary-augmented MoE We use MMD (Gretton et al., 2012) as the adversary to minimize the divergence between the marginal distribution of target domain and source domains.
at each training epoch, given the K batches {x S 1 , x S 2 , ..., x S K } from all the source domains, we sample a batch (unlabeled) x T from our target domain, and minimize the MMD: where measures the discrepancy between D S and D T based on Reproducing Kernel Hilbert Space (RKHS).