The model fine tuned on a QA and LM objective is rated as significantly worse by human annotators, despite achieving higher scores in the automatic metrics.
the training objective given by these reward sources does not correspond to true question quality, despite them being intuitively good choices.