One solution to this problem is to approximate Q(s, a) using a parametrized function Q(s, a; θ), which can generalize over states and actions by considering higher-level attributes (Sutton and Barto, 1998;Branavan et al., 2011a).
creating a good parametrization requires knowledge of the state and action spaces.