Task success combined with dialogue cost can be linked to human judgments like user satisfaction via the PARADISE framework (Walker et al., 1997).
in chitchat tasks, which we study in this work, automatic metrics and their relation to human ratings are less well-understood.