We introduce various phrase granularity over the conventional word-level memory to generate phrase level memory.
we first transform the input layer H to a phrase level memory by function F h in certain attention head: where H g is the generated phrase level memory, h denotes the h-th head which is used to generate a certain granularity of phrase memory, and F h is a representation function with its own trainable parameters.