Formally, we write the step-wise reward s(y t |y 1:t 1 , Y ⇤ ) as follows: where N (Y,Ỹ ) represents the occurrence of sub-sequenceỸ in whole sequence Y .
if a certain sub-sequence y t n+1:t from Y appears less times than in the reference Y ⇤ , y t receives reward.