3 We use multi-class logistic regression to fit the layer-wise hidden representation learned by a well-trained 6-layer Transformer (Vaswani et al., 2017) over each training instance x, y <t .
given a context x, y <t , for each task Y k and a hidden representation h l (x, y <t ) of this context, which is fixed as constant, we predict the cluster Y k (y t ) according to the following probability: where θ l Y k is the parameter of the logistic regression model for task Y k at l th layer.