We propose an oracle setup that can be considered as an upper bound to what can be achieved with a learned policy.
we use an oracle policy that is allowed to always pick a subset of examples that maximizes its target metric on a development set, which has the same distribution as the test set.