A promising result for language learning has been shown in (Yu and Siskind, 2013) and a quite challenging effort to describe cooking activities was made in (Regneri et al., 2013).
these studies rely only on visual information, while we aim to build a system that is able to describe everyday activities using multimodal information.