In the end, "the instructions described the task only through [a handful of] examples of relatedness" (Marelli et al., 2014).
of this hands-off annotation guideline, the SICK dataset contains glaring inconsistencies in semantic relatedness scores, even for sentence pairs where the only difference between two sentences is due to the same known transformation.