Despite not having access to syntactic structure, TRANSFORMER-SINGLE far outperforms all surface cues above.
we find evidence for the claim that Transformer models implicitly learn language understanding relevant to coreference resolution.