String-similarity metrics against a reference are known to be partial and coarsegrained aspects of the task (Callison-Burch et al., 2006), but are still the common practice in various text generation tasks.
their opaqueness and difficulty to interpret have led to efforts to improve evaluation measures so that they will better reflect the requirements of the task (Anderson et al., 2016;Sulem et al., 2018;Choshen and Abend, 2018b), and to increased interest in defin-ing more interpretable and telling measures (Lo and Wu, 2011;Hodosh et al., 2013;Choshen and Abend, 2018a).