The best performance has been consistently obtained for mini-batch sizes between m=2 and m=32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance. He cited this paper which has just been posted on arXiv few days ago (Apr 2018), which is worth reading,ĭominic Masters, Carlo Luschi, Revisiting Small Batch Training for Deep Neural Networks, arXiv:1804.07612v1 Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. More importantly, it's bad for your test error.įriends dont let friends use minibatches larger than 32. Training with large minibatches is bad for your health. I just saw this comment on Yann LeCun's facebook, which gives a fresh perspective on this question (sorry don't know how to link to fb.) Escaping From Saddle Points-Online Stochastic Gradient for Tensor Decomposition. Large-scale machine learning with stochastic gradient descent. The minibatch methodology is a compromise that injects enough noise to each gradient update, while achieving a relative speedy convergence.ġ Bottou, L. The disadvantage is it's terribly inefficient and you need to loop over the entire dataset many times to find a good solution. However, the noisiness is exactly what you want in non-convex optimization, because it helps you escape from saddle points or local minima(Theorem 6 in ). Since it's based on one random data point, it's very noisy and may go off in a direction far from the batch gradient. In pure SGD, on the other hand, you update your parameters by adding (minus sign) the gradient computed on a single instance of the dataset. But the real handicap is the batch gradient trajectory land you in a bad spot (saddle point). In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent 1.

Giant machines 2017 trainer