For our application of counterfactual learning to human bandit feedback, we found another deviation from standard counterfactual learning to be helpful: For humans, it is hard to assign a graded reward to a query at a sequence level because either the query is correct or it is not.
with a sequence level reward of 0 for incorrect queries, we do not know which part of the query is wrong and which parts might be correct.