In particular, deeper or more complex models (which require greater time and resources to train) generally perform best in the supervised setting, whereas shallow log-bilinear models work best on unsupervised benchmarks.
skipThought Vectors (Kiros et al., 2015) perform best on the majority of supervised evaluations, but sDAEs are the top performer on paraphrase identification.