Similar to RNN-RBM (Boulanger-Lewandowski et al., 2012), we constrain RNN hidden units (u (t) ) to convey temporal information, while RSM hidden units (h (t) ) to model conditional distributions.
parameters (b v (t) , b h (t) ) are time-dependent on the sequence history at time t (via a series of conditional RSMs) denoted by Θ (t) ≡ { V (τ ) , u (τ ) |τ < t}, that captures temporal dependencies.