To tackle this problem, we have developed an approach to jointly process language and vision by incorporating semantic role information.
we use a benchmark dataset (TACoS) which consists of parallel video and language descriptions in a complex cooking domain (Regneri et al., 2013) in our investigation.