Unlike measuring embedding bias, measuring classification bias is difficult: most NLP datasets are not annotated with protected attributes, precluding the use of standard fairness measures such as equal opportunity (Hardt et al., 2016).
Equalized odds requires that both true and false positive rates be equal across groups (Hardt et al., 2016).