QGen employs an LSTM (LSTM q ) to generate the token sequence for each question conditioned on h t , which is used to initialise the hidden state of LSTM q .
as input at every time step, QGen receives a dense embedding of the previously generated token w i−1 and the image features I: We optimise QGen by minimising the Negative Log Likelihood (NLL) of the human dialogues and use the adam optimiser (Kingma and Ba, 2015): in our architecture the LSTM q of QGen in combination with the LSTM e of the Encoder form a sequence-to-sequence model (Sutskever et al., 2014), conditioned on the visual and linguistic context -in contrast to the baseline model, where question generation is performed by a single LSTM on its own.