Previous studies have mostly focused on estimating each annotator's overall reliability on the entire annotation task.
when constructing models that learn from noisy labels produced by multiple annotators, it is important to accurately estimate the reliability of annotators.