In practice, this issue often goes unnoticed because each word in the attribute set, at least for gender association, has a counterpart that appears with roughly equal frequency in most training corpora (e.g., 'man' vs. 'woman', 'boy' vs. 'girl').
this is not guaranteed to hold, especially for more nebulous attribute sets (e.g., 'pleasant' vs. 'unpleasant' words).