We set the number of hidden units proportional to the input length (see Section 3.1 for configuration details).
we compute each attention score a att ij as follows: where att ∈ {w, p, m} is an attention type, W att and W att2 are the weights of the first and second network layers, b att and b att2 are their respective biases, s att ij is an intermediary representation, and σ is a sigmoid activation function.