Based on this understanding, we define a new expected reward function R APRIL for policy π as follows: where Note that ∆ x (y i , y j ) equals 1 if y j is preferred over y i and equals 0 otherwise (see §3.1).
r(y|x) is the relative position of y in the (ascending) sorted Y(x), and it can be approximated by preference learning.