As shown in Figure 2, different from a stacked two-layer LSTM, the high-level LSTM here operates at a lower temporal resolution and runs one step every s time steps.
in formula, o e H j , h e H j = e H (f e H j , h e H j−1 ) , where e H denotes the high-level LSTM whose output and hidden state at j are o e H j and h e H j .