Thus, we use the development set to find the optimized thresholds.
we calculate a score for each Type 1 candidate in the development set and find the thresholds that maximize the F 1 score of detecting Type 1 spoonerism on this set.