Define analogous estimates based on the k-th minibatch bŷ If l N denotes an objective function computed from all N samples in the training data and l k denotes the same objective computed from B k , a condition on the correctness of SGD is that the gradient of l k (with respect to model parameters) is consistent with the gradient of l N on average: where denotes the bias of the stochastic gradient estimator.
any loss of the form l N = (1/K) k l k that decomposes over independent minibatches (e.g., the cross-entropy loss for supervised classification) satisfies (8) with = 0.