The information distillation happens in the recaller component where only the most important information is preserved.
the recaller outputs fewer vectors s than that of the input Figure 1: Our cascaded attention based unsupervised information distillation framework.