One main reason is that deep networks could easily fit any task partitions if we optimize on training loss only, making the learned metrics not generalize, as discussed in Section 6.
this work adopts a pipeline training approach and employing validation sets for task clustering.