As a baseline, we trained an LSTM tagger with a CRF output layer to predict targets consisting of the simple Cartesian product of symptom body systems and their status.
when two scribes pick different valid labels and the third misses them, the "voted" reference is not better than the single reference.