Pre-trained Word Embeddings Utilizing word embeddings pre-trained on large unlabeled text has shown to be one of the most effective ways to increase performance on various NLP tasks.
starting from a small batch size and increasing it during training would result in a more global, but sharp minimizer.