In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF.
we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences.