3 To get a sense of the relative need for lexical normalisation, we perform analysis of the distribution of OOV words in different text types.
we calculate the proportion of OOV tokens per message (or sentence, in the case of edited text), bin the messages according to the OOV token proportion, and plot the probability mass contained in each bin for a given text type.