Increasing the number of clusters introduces smaller clusters for which it is difficult to estimate the noise matrix, due to the limited training resources.
decreasing the number of clusters can generalize too much, resulting in loss of information on the noise distribution.