Also, to avoid affecting previously learned  Step-2 knowledge, we constrain θ 1,k and ψ 2,k to be orthogonal (Condition 2.2).
strictly imposing this condition into the objective function is not feasible (Bousmalis et al., 2016), hence we add a penalizing term into the objective function as an approximation to the orthogonality condition: Both Condition 2.1 and 2.2 are mutually dependent, because for two matrices' product to be zero, they share basis vectors between them, i.e., for an n-dimensional space, there are n basis vectors and if p of those vectors are assigned to one matrix, then the rest of the n − p vectors (or subset) should be assigned to the other matrix.