We suspect that a significant portion of this degradation is due to distribution shift (as evidenced by improvement in Hindi F 1 in Parallel regime).
one promising route to improving our models might be to incorporate domain adaptation techniques, which aim to build classifiers robust to various forms of distribution shift.