To verify this, we conducted a separate human evaluation wherein we asked 15 in-house evaluators to read conversations (without the background resources) from our dataset and rate them on five different parameters.
they were asked to check if the conversations were 1) intelligible: i.e., an average reader could understand the conversation 2) coherent: i.e., there were no abrupt context switches 3) grammatically correct 4) on-topic: i.e., the chat revolved around the concerned movie with digression limited to related movies/characters/actors and 5) natural two-person chats: i.e., the roleplay setup does not make the chat look unnatural.