Although a large-width beam search generates responses with higher probability and diversity, increasing beam width proportionally slows down the inference process.
to retrieve better responses and meanwhile maintain the inference efficiency, we propose Prospective-Performance Network to estimate the future rewards of partial responses in the inference procedure of NRG.