We argue that to do proper temporal reasoning, pre-determined, query independent context features may not cover all possible temporal relations.
we propose to model the context as a latent variable, allowing our method to learn which context moments to consider as a function of the video and importantly, the query.