Our intuition is that low entropy words are indicative of semantically coherent topics and therefore more informative, while high entropy words indicates words whose usage is more topical diverse and therefore less informative.
the entropy of word w given the class label c is defined as follows: where C denotes the number of classes (in our case violent and non-violent) and p(w|sd c i ) denotes the probability of word w given the document sd i in class c. to the general E SD , the class word entropy characterises the usage of a word in a particular document class.