Previous best classification results on TREC data is achieved by SVM using unigrams, bigrams, whword, head word, POS tags, hypernyms, WordNet synsets and a bunch of hand-coded rules.
the forget gate is able to decay the information according to the context rather than a fixed decaying weight in tensor product based CNNs.