The replacement procedure is the same as the one described for MUR, where a uniform random index t is selected in the dialog context and u t is replaced by a negative sample q.
while MUR strives to create a model that finds the original utterance, given the replacement index t, InI aims to train a model that can identify the replacement position t. this is done via:û Finally, the loss function is to minimize the cross entropy of the replaced index: This pretraining objective aims to explicitly model the coherence of the dialog, which encourages both local representations of each individual utterance and a global representation of the dialog context.