To address this need, we have developed an approach on learning procedural tasks jointly from language instruction and visual demonstration.
we use And-Or Graph (AoG), which has been used in many computer vision tasks and robotic applications (Zhao and Zhu, 2013;Li et al., 2016;, to represent a hierarchical task model that not only captures symbolic concepts (extracted from language instructions) but also the corresponding visual state changes from the physical environment (detected by computer vision algorithms).