In fact, the short-list approach in (Schwenk, 2007) and the adaptive importance sampling in (Bengio and Senecal, 2008) have exactly this intuition.
in the multi-class setup, subsampling like this has to be very careful.