Formally, given a set of tokens T and two accompanying system responses S 1 and S 2 , we quantify how many of the tokens t i ∈ T have a different label in S 1 and S 2 : Note that switching the inequality condition ( =) to equality (=) actually yields the accuracy metric.
taking S 1 as the key and the S 2 as the response and calculating accuracy produces the inverted results of our metric, i.e.