In our qualitative analysis, we observed that randomness in dialogues might hold a clue for improving evaluation scores.
we measured the perplexity of each system output using dialogue-act trigrams and obtained 72.8 for "Random," 27.4 for "Human dialogue," 7.4 for "POMDP," 3.2 for "HMM," 2.5 for "Even-POMDP," and 1.7 for "Rule-based."