Previous works (She et al., 2014;Misra et al., 2015;Chai, 2016, 2017) explicitly model verb semantics as desired goal states and thus linking natural language commands with underlying planning systems for action planning and execution.
these studies were carried out either in a simulated world or in a carefully curated simple environment within the limitation of the robot's manipulation system.