Earlier, I discussed how I had no luck using second-order optimization methods on a convolutional neural net fitting problem, and some of the reasons why stochastic gradient descent works well on this class of problems.

Stochastic gradient descent is not a plug-and-play optimization algorithm; it requires messing around with the step size hyperparameter, forcing you to expend a lot of energy getting the optimization to work properly, time probably better spent considering different model forms or novel analyses. Since deep neural nets have become very popular, a lot of research has gone into eliminating learning rates within the context of stochastic gradient descent.

Adagrad is a theoretically sound method for learning rate adaptation which has has the advantage of being particularly simple to implement. The learning rate is adapted component-wise, and is given by the square root of sum of squares of the historical, component-wise gradient. Pseudo-code:

``` master_stepsize = 1e-2 #for example fudge_factor = 1e-6 #for numerical stability historical_grad = 0 w = randn #initialize w while not converged: E,grad = computeGrad(w) historical_grad += g^2 adjusted_grad = grad / (fudge_factor + sqrt(historical_grad)) w = w - master_stepsize*adjusted_grad ```

Simple enough, right? Empirically, adagrad works much better – i.e. convergence is faster and more reliable – than simple SGD when the scaling of the weights is unequal. It is also not very sensitive to the master step size; just find some value which converges in a reasonable amount of time and leave it as is.

Adagrad has the natural effect of decreasing the effective step size as a function of time. Perhaps you have good reason to use your own step-size decrease schedule. In this case, you can use a running average of the historical gradient instead of a sum. Pseudocode:

``` autocorr = .95 #for example master_stepsize = 1e-2 #for example fudge_factor = 1e-6 #for numerical stability historical_grad = 0 w = randn #initialize w while not converged: E,grad = computeGrad(w) if historical_grad == 0: historical_grad = g^2 else: historical_grad = autocorr*historical_grad + (1-autocorr)*g^2 adjusted_grad = grad / (fudge_factor + sqrt(historical_grad)) w = w - master_stepsize*adjusted_grad ```

1. kevin says: