Most human evaluation studies for (interactive) semantic parsers so far (Chaurasia and Mooney, 2017;Gur et al., 2018;Yao et al., 2019) use pre-existing test questions (e.g., from datasets like WikiSQL).
this introduces an undesired discrepancy, that is, human evaluators may not necessarily be able to understand the true intent of the given questions in an faithful way, especially when the question is ambiguous, vague, or containing unfamiliar entities.