We propose the task of "sequencing" in a set of image-caption pairs, with the motivation of learning temporal common sense.
we experiment with 2 networks -(1) A language-alone unary model (Skip-Thought+MLP) that uses a Gated Recurrent Unit (GRU) proposed by Cho et al.