D-1, D-2, D-U: The distinctiveness denotes the number of unique unigrams, bigrams, and utterances normalized by each total count (Li et al., 2016a;Xu et al., 2018).
a feed-forward network takes the current context RNN's output c t as the query and all previous context outputs from both context RNNs as keys and values to compute attention scores.