(Venkatesh et al., 2018) introduced a framework for evaluating the quality of the conversations based on topical diversity, coherence, engagement and conversational depth and showed that these metrics conform with human evaluation.
a big part of their metrics relies on human labels, which makes the evaluation system not scalable.