Given an image x ∈ D x u , we retrieve a caption in the unpaired dataset, i.e.,ỹ ∈ D y u , that has the highest score obtained by the discriminator, i.e., the most likely caption to be paired with the given image as vice versa for unpaired captions: By this retrieval process over all the unpaired dataset, we have image-caption pairs {(x i , y i )} from the paired data and the pairs with pseudolabels {(x j ,ỹ j )} and {(x k , y k )} from the unpaired data.
these pseudo-labels are not noisefree, thus treating them equally with the paired ones is detrimental.