The probability of a pair of tokens a and b and a pair of respective languages a and b is given by the product of the probabilities Thus, the sum of the probabilities of all pairs of language pairs is equal to 1, and we can efficiently compute the probability that the pair of tokens are in different languages as: where P L ( | x) is once again the probability that token x is in language , according to a character-based model, and L is the set of all languages considered.
given a tweet, our model attempts to find a pair of words where P( a = b | a, b) is higher than 0.95.