Formally, a MDP is a list [Z, A, T, R], where Z = {z} is a set of states, A = {a} is a set of actions, T (z, a) is a transition function, which determines the next state z = T (z, a) after performing action a on z, and R is a reward function.
we describe each in detail below: States: Each state z is a list of lists:  at each state, we have n candidate concepts v 1 ...v n .