Therefore, we approximate the performance of a supervised system by clustering instances according to their gold standard role after introducing some noise.
we randomly selected 5% of the gold standard roles and mapped them to an erroneous role.