We also tried to fine-tune the BERT to directly generate the restored utterance, but the result is far from satisfactory.
we directly append selected words to the original input sequence as extra guidance for generating restored sentences.