We use beam size of 5, and for all models trained with cross entropy, it outperforms lower beam sizes on CHAIR.
when training models with the self-critical loss, beam size sometimes leads to worse performance on CHAIR.