, x I }, each hidden state in the l-th layer is constructed by attending to the states in the (l − 1)-th layer.
1 the (l − 1)-th layer H l−1 ∈ R I×d is first transformed into the queries Q ∈ R I×d , the keys K ∈ R I×d , and the values V ∈ R I×d with three separate weight matrices.