Intuitively, such disagreements among humans should be allowed because different annotators might have different subjective views of the world and might think differently when they encounter the same reasoning task.
from a descriptive perspective, evaluating the capacity of NLP models in predicting not only individual human opinions or the majority human opinion, but also the overall distribution over human judgments provides a more representative comparison between model capabilities and 'collective' human intelligence.