Also concurrent, SpanBERT (Joshi et al., 2019), another self-supervised method, pretrains span representations achieving state of the art results (Avg.
yet, both variants perform much worse with longer context windows (512 tokens) in spite of being trained on 512-size contexts.