This is a Matlab implementation of the Adam optimiser from Kingma and Ba [1], designed for stochastic gradient descent. It maintains estimates of the moments of the gradient independently for each parameter.
fmin_adam is an implementation of the Adam optimisation algorithm (gradient descent with Adaptive learning rates individually on each parameter, with Momentum) from Kingma and Ba [1]. Adam is designed to work on stochastic gradient descent problems; i.e. when only small batches of data are used to estimate the gradient on each iteration, or when stochastic dropout regularisation is used [2].
Examples
###Simple regression problem with gradients
Set up a simple linear regression problem: , where . We'll take for this example. Let's draw some samples from this problem:
Now we define a cost function to minimise, which returns analytical gradients:
function [fMSE, vfGrad] =LinearRegressionMSEGradients(phi, vfInput, vfResp)
% - Compute mean-squared error using the current parameter estimate
vfRespHat =vfInput.*phi(1) +phi(2);
vfDiff =vfRespHat-vfResp;
fMSE =mean(vfDiff.^2) /2;
% - Compute the gradient of MSE for each parametervfGrad(1) =mean(vfDiff.*vfInput);
vfGrad(2) =mean(vfDiff);
end
Initial parameters phi0 are Normally distributed. Call the fmin_adam optimiser with a learning rate of 0.01.
Define the function to minimise; in this case, the mean-square error over the regression problem. The iteration index nIter defines which mini-batch to evaluate the problem over.
fun is a function handle [fCost <, vfCdX>] = @(x <, nIter>) defining the function to minimise . It must return the cost at the parameter x, optionally evaluated over a mini-batch of data. If analytical gradients are available (recommended), then fun must return the gradients in vfCdX, evaluated at x (optionally over a mini-batch). If analytical gradients are not available, then complex-step finite difference estimates will be used.
To use analytical gradients (default), set options.GradObj = 'on'. To force the use of finite difference gradient estimates, set options.GradObj = 'off'.
fun must be deterministic in its calculation of fCost and vfCdX, even if mini-batches are used. To this end, fun can accept a parameter nIter which specifies the current iteration of the optimisation algorithm. fun must return estimates over identical problems for a given value of nIter.
Steps that do not lead to a reduction in the function to be minimised are not taken.
Output arguments
x will be a set of parameters estimated to minimise fCost. fval will be the value returned from fun at x.
exitflag will be an integer value indicating why the algorithm terminated:
0: An output or plot function indicated that the algorithm should terminate.
1: The estimated reduction in 'fCost' was less than TolFun.
2: The norm of the current step was less than TolX.
3: The number of iterations exceeded MaxIter.
4: The number of function evaluations exceeded MaxFunEvals.
output will be a structure containing information about the optimisation process:
`.stepsize` — Norm of current parameter step
`.gradient` — Vector of current gradients evaluated at `x`
`.funccount` — Number of calls to `fun` made so far
`.iteration` — Current iteration of algorithm
`.fval` — Value returned by `fun` at `x`
`.exitflag` — Flag indicating reason that algorithm terminated
`.improvement` — Current estimated improvement in `fun`
The optional parameters stepSize, beta1, beta2 and epsilon are parameters of the Adam optimisation algorithm (see [1]). Default values of {1e-3, 0.9, 0.999, sqrt(eps)} are reasonable for most problems.
The optional argument nEpochSize specifies how many iterations comprise an epoch. This is used in the convergence detection code.
The optional argument options is used to control the optimisation process (see optimset). Relevant fields:
[2] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. "Improving neural networks by preventing co-adaptation of feature detectors." arXiv preprint. https://arxiv.org/abs/1207.0580
请发表评论