While this work is very preliminary, and the experimental setting somewhat unrealistic (dense features artificially generated), especially for many natural language tasks, the results are dramatic enough to be worth noting.
minka found that a version of conjugate gradient descent worked extremely well -much faster than GIS.