Data parallelism and model parallelism are two commonly used techniques to improve the speed of training .
multiple GPUs or T-PUs 1 are needed which requires additional replication and combination costs.