The right part in Figure 1 shows details of the Sub-layerexpanded Transformer.
the equation of multi-head attention computing procedure is as following: where W i Q , W i K and W i V are the parameter matrices, Q, K and V indicates the query, key and value representations.