Therefore, our second idea is to apply an utterance drop regularization to effectively regularize the hierarchical RNNs, in order to facilitate the use of latent variables.
at each time step, the utterance encoder vector h enc t is randomly replaced with a generic unknown vector h unk with a probability p. This regularization weakens the autoregressive power of hierarchical RNNs and as well alleviates the data sparsity problem, since it induces noise into the context vector h cxt t which conditions the decoder RNN.