Thus, it is desirable to have a diverse distribution of the matching behaviors n=1 , we propose a diversity loss L div to strengthen the matching behaviors between reliable instance-sentence pairs and penalize the unreliable ones during training.
we first normalize {s(q, p n )} N n=1 by softmax and then penalize the entropy of the distribution of {s (q, p n )} N n=1 by defining the diversity loss as Note that the smaller L div is, the more diverse {s(q, p n )} N n=1 will be, which implicitly encourages the matching scores of semantically aligned instance-sentence pairs being larger than those of the misaligned pairs.