We alter a LM to a label-conditional LM, i.e., for position i in sentence S with label y, we aim to calculate p τ (•|y, S\{w i }) instead of the default p τ (•|S\{w i }) within the model.
we concatenate each embedded label y with a hidden layer of the feed-forward network in the bidirectional LM, so that the output is calculated from a mixture of information from both the label and context.