We hypothesise that a system that encodes gaze data as a proxy for human visual attention will lead to better, more human-like descriptions.
we propose that training such a system with eye-movements sequentially aligned with utterances (see Figure 1) will produce descriptions that reflect the complex coordination across modalities observed in cognitive studies.