This type of linguistic reasoning is critical for interactions grounded in visually complex environments, such as in robotic applications.
commonly used resources for language and vision (e.g., Antol et al., 2015;Chen et al., 2016) focus mostly on identification of object properties and few spatial relations (Section 4; Ferraro et al., 2015;Alikhani and Stone, 2019).