However, this notion of depth likely does not involve hierarchical processing of the data: across different time steps, we repeatedly apply the same transformation to compute the memory contribution of the input (W ), to compute the response value from the current memory (U ) and to compute the next memory vector from the previous one (V ).
assuming the input vectors {x t } together lie in the same representation space, as do the output vectors {y t }, hidden representations {h t } lie in the same space as well.