Similarly, (Yao et al., 2019) relies on a pre-defined two-level hierarchy among components in an If-Then program and cannot generalize to formal languages with a deeper structure.
a more direct comparison of various settings under the same average number of questions can be found in appendix C. To better understand how each kind of error detectors works, we investigate the portion of questions that each detector spends on right predictions (denoted as "Q r ").