Based on above triple sampling strategies, we define the task of inconsistent order detection as: given a dialogue history and the target utterance pair (Q t , A t ) for evaluation, the model needs to predict whether the sampled triple is misordered (output: 1).
1 e.g., for the added padding utterance Q−2, it is represented as a sequence of one same padding word {w where N is the roundedup averaged length of utterances in the dataset.