After that, we generate each stem σ 0 , prefix σ − , and suffix σ + independently: Token-POS Model This model captures the dependencies between the syntactic categories of adjacent tokens with a first-order HMM.
conditioned on the type-level assignments, we generate (unsegmented) tokens w and their POS tags t: where the parameters of the multinomial distributions are generated by Dirichlet priors: Here, W t refers to the set of word types that are generated by tag t. conditioned on tag t, we can only generate word w from the set of word types in W t which is generated earlier (Lee et al., 2010).