The standard measure of inter-annotator agreement in various natural language labelling tasks is Cohen's kappa (Cohen, 1960).
this coefficient is designed to measure agreement between two annotators only.