(2014) use automatic suggestions for de-identification of medical texts and find no change in inter-annotator agreement or annotation time.
to these works, we use a control group of two annotators, who never receive suggestions, and compare the performance of all annotators to previous annotations they performed without annotation suggestions.