We have investigated the use of AL in a real-life annotation experiment with human annotators instead of traditional simulations with gold labels for named entity recognition and sentiment classification.
aL depends on accurate assessments of uncertainty and informativeness and such an accurate assessment is made more difficult if labels are noisy as is the case in crowdsourcing.