One approach for achieving robustness to the number of observations is through the training procedure.
when computing the approximate posterior for a particular font in our training corpus, we mask out a randomly selected subset of the characters before passing them to the encoder.