This simplest distribution approximation does not give an improvement in our experiments because of the huge information loss.
we propose a better approach of sample selection by clustering over the embedding space from the model, and choose one representative from each cluster to store in the memory.