For each dialog, a ranking score is calculated as: In the training phase, the human ratings are used to set α by minimizing the loss function: The eval function returns 0 if (x, y) pair is ranked correctly, and 1 otherwise.
lOSS score is the percentage of misordered pairs where the order of the predicted scores disagree with the order indicated by human judges.