This is because the document encoder, SpanBERT, dominates the memory usage during inference (as also observed by Xia et al., 2020).
the peak memory usage during inference is determined by the mention proposal stage rather than the mention clustering stage.