Our notion of relevance is closely related to coherence; when thinking of judging a continuation, we believed that it would be more natural for annotators to judge the relevance of the continuation to its context, rather than judging the coherence of the resulting story.
coherence is a property of a discourse, while relevance is a property of a continuation (in relation to the context).