To demonstrate empirically that the learned states indeed align with field types, we calculate the average purity of the discrete states learned for both datasets in Table 5.
for each discrete state for which the majority of its generated words appear in some r j , the purity of a state's record type alignment is calculated as the percentage of the state's words that come from the most frequent record type the state represents.