The MLP-1 probe is defined as follows: Note here that h i broadcasts to R T ×d ,while W 1 ∈ R ×d , and W 2 ∈ R 1× broadcast as well.
each [h j ; h i ] pair is mapped to a single scalar independently of all others, leading to T logits used as input to the softmax.