State s is comprised of the state of the external environment E, and the state of the document (d, W ), where W is the list of all word spans mapped by previous actions.
an action a selects a span W a of unused words from (d, W ), and maps them to an environment command c. of a, the environment state changes to E ∼ p(E |E, c), and the list of mapped words is updated to W = W ∪ W a .