R D , and then propose to add random dropout noise into the embedding input to simulate the dropout process as follows: where OneHot is a function to transform a word x into its one-hot form OneHot(x) ∈ R V , and  are given as the evidence, we first define an objective function as L(f θ (x), y) and then derive its lower bound as follows (withp = 1 − p): where P(b) is the prior distribution, and Bern(p) denotes the Bernouli approximate posterior with parameter p. Here we use E(x) as the simplied we separate the text classification model's parameters θ with the embedding parameters W and assume the classification model f θ directly takes embedding E as input.
the Bernouli distribution is hard to reparameterize, where we need to enumerate 2 V different values to compute the expectation over the stochastic dropout vector b.