This corresponds to measuring an expected reduction in entropy, i.e., uncertainty associated with a random factor.
the information gain G ∈ R for a gappy-phrase g is defined as where H(C) represents the entropy of the classification, C is a stochastic variable taking a class, X g is a stochastic variable representing the presence (X 1 g ) or absence (X 0 g ) of the phrase g, P (X g ) represents the probability of presence or absence of the phrase g, and H(C|X g ) is the conditional entropy due to the phrase g. We use top-k phrases based on the information gain G. we use the top 40% of phrases to compute the feature values.