For the related tasks of document and multidocument summarization, Graham (2015) provides a fine-grained comparison of automated evaluation methods.
to the best of our knowledge, no studies of automatic evaluation metrics exist for abstractive compression of shorter texts.