This is inspired by common practice in machine translation.
we introduce a vector h s 0 , which is fixed to be a vector of 0s of dimension d. This h s 0 represents NULL and is used with other h s j to derive the attention vectors {a k } N k=1 .