Table 5: A distance matrix for all pairs of probability distributions over POS-tags shown in Table 4 and the uniform distribution; the BC-(or Hellinger) distance (Bhattacharyya, 1943;Nikulin, 2002) between discrete distributions p and q (over x ∈ X ) ranges from zero (iff p = q) to one (iff p • q = 0, i.e., when they do not overlap at all).
(ii) Punctuation at sentence boundaries further helps distinguish full sentences from fragments like headlines and titles, allowing us to model grammatical differences between complete and incomplete sentences.