I.e., the loss of each model is minimized independently by separate optimizers with their own hyperparameters.
it is in some sense a multitask learning model and we must define a schedule in which individual models are updated.