In our companion paper, (Iyyer et al., 2017), we used a simple form of approximated reward to get feedback as early as possible during search.
the semantic parse will be executed as soon as the parse is executable (even if the parse is still not completed) during search.