In SS and DSS, the probability of using reference words s is annealed using inverse sigmoid decay : s = k/(k + exp(i/k)) at the i-th checkpoint with k = 10.
we train the models using the Adam optimizer (Kingma and Ba, 2015) with a batch size of 1024 words.