Thus the value network of the actor-critic model fails to estimate good value of the given state since deleting a good frame will not result in the loss of QA accuracy.
we compute the reward R (t) as the accuracy difference between at time step t and t − 1 then use only the policy with non-episodic REINFORCE for training.