In our experiments, we set the threshold to 0, PMI score lower than it will result in a segmentation.
for each dataset, we use PMI to perform unsupervised segmentation and collect the segmented words from it to build the n-gram lexicon N .