We analyse their role through a visualizasion of the activation in the hidden layer of the bi-LSTM.
we approximated the objective function through a linear function, and estimated the contribution of each word to the true class probability by computing the prime derivative of the output scores with respect to the embeddings.