We test several normalisation strategies: • by batch size (number of training instances) • by (target) sentence length • without normalisation (longer sentences were always assigned a higher perplexity) Experimental results show no significant performance differences for any of the strategies.
multiple GPUs or T-PUs 1 are needed which requires additional replication and combination costs.