Unlike measuring embedding bias, measuring classification bias is difficult: most NLP datasets are not annotated with protected attributes, precluding the use of standard fairness measures such as equal opportunity (Hardt et al., 2016).
as shown in section 3.3, datasets currently used to estimate classification bias in NLP – such as WinoBias (Zhao et al., 2018b) and WinoGender (Rudinger et al., 2018) – are too small to conclusively identify bias except in the most egregious cases.