This task has been reproduced with variations on the language (Cholakov et al., 2014;Fabre et al., 2014) or the size of the dataset (Kremer et al., 2014).
by beginning to identify the main differences in terms of substitutes proposed by humans and NLP systems, we can complete the initial analysis proposed by (Tanguy et al., 2016) who found that there are also important differences in the difficulty encountered for specific target sentences.