One problem with accuracy as a scoring function is that these classification problems are often very imbalanced.
for very rare words, a high accuracy might not necessarily imply that the corresponding direction is accurate.