Despite the great success of the above models, they are designed for sentence-level transla-Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc, China.
the equation of multi-head attention computing procedure is as following: where W i Q , W i K and W i V are the parameter matrices, Q, K and V indicates the query, key and value representations.