For our experiments, we train all our models on a single pass of the May 2017 dump of the English Wikipedia with lowercased unigrams.
this differentiability comes at the cost of drawing N Gumbel samples per each categorical sample, where N is the number of categories.