We use a small amounts of doublyannotated Twitter data to estimate F 1-scores and confusion probabilities, and incorporate them during training via a modified loss function.
we use POS annotations made by two annotators on a set of 500 newly sampled tweets to estimate our agreement scores, and train models on existing Twitter data sets (described below).