We observe that before finetuning, the attention patterns on [SEP] tokens and periods is almost identical between sentences.
in this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF.