We evaluate individual responses at each question in a retrieval setting according to (Das et al., 2017).
the dialog agent is given a list of 100 candidate answers of each question and asked to rank the list.