However, conditioning on the context makes the range of training target x t very sparse; even in a large-scale conversation corpus such as Ubuntu Dialog (Lowe et al., 2015), there exist one or very few target utterances per context.
hierarchical RNNs, given their autoregressive power, can easily overfit to training data without using the latent variable.