In this work, we propose holistic metrics that evaluate distinctive aspects of generated dialogues.
we consider (1) context coherence of a dialogue: the meaningfulness of a response within the context of prior query, (2) language fluency of generated responses: the quality of phrasing relative to a human native speaker, (3) response diversity of a set of generated sentences: the variety in meaning and word choice of responses, and (4) logical self-consistency: the logical consistency of utterances from a dialogue agent.