Training with the combined labeled and unlabeled data, the objective is to maximize the mixed likelihood: Since D ′ contains much more instances than D (1.7M vs. 40K for English, and 4M vs. 16K for Chinese), it is likely that the unlabeled data may overwhelm the labeled data during SGD training.
we propose a simple corpus-weighting strategy, as shown in Algorithm 1, where is the subset of training data used in k th update and b is the batch size; η k is the update step, which is adjusted following the simulated annealing procedure (Finkel et al., 2008).