The possible explanation is that domains within the same group are more close.
adapting over within group tasks is easier than adapting over cross group tasks, if without any domain adaptation regularizer.