Since Baseline 1 barges-in on users after an information peak, when ID may still be high, it continuously receives a negative reward reflecting the user preference for late barge-ins.
the goal of each SMDP is to find an optimal policy π * that maximises the reward for each visited state, according to where Q i j (s, a) specifies the expected cumulative reward for executing action a in state s and then following π * .