AdpBP90b Class Reference

Offers the second version of the gradient descent based optimization algorithm with individual adaptive learning rates by Silva and Almeida (Adaptive BackPropagation). This optimization algorithm introduced by Silva and Almeida adds individual adaptive learning rates and weight-backtracking to standard steepest descent. More...

#include <AdpBP.h>

Inheritance diagram for AdpBP90b:

List of all members.

Public Member Functions
void	init (Model &model)
	Initializes the optimizer with default parameters.
void	initUserDefined (Model &model, double increaseFactorP=1.2, double decreaseFactorP=0.7, double alphaP=0.5, double eta0P=0.1)
	Initializes the optimizer with user defined parameters.
double	optimize (Model &model, ErrorFunction &errorfunction, const Array< double > &input, const Array< double > &target)
	Performs a run of the optimization algorithm.
void	clearMemory ()
	Resets internal variables of the class instance.
Protected Attributes
double	increaseFactor
	The increase factor for the adaptive learning rate.
double	decreaseFactor
	The decrease factor for the adaptive learning rate.
double	alpha
	The momentum factor $\alpha$ that controls the effect of the momentum term.
double	eta0
	The initial value for the adaptive learning rate $\eta$ for all weights of the network.
double	oldError
	The old error value.
Array< double >	deltaw
	The final update values (including the individual learning rates) for all weights.
Array< double >	v
	The update values (without the individual learning rates) for all weights.
Array< double >	dedwOld
	The last error gradient.
Array< double >	eta
	The adaptive learning rate $\eta$ (for each weight individually).

Detailed Description

Offers the second version of the gradient descent based optimization algorithm with individual adaptive learning rates by Silva and Almeida (Adaptive BackPropagation). This optimization algorithm introduced by Silva and Almeida adds individual adaptive learning rates and weight-backtracking to standard steepest descent.

: To avoid bad choices of the learning rate, the algorithm by Silva and Almeida uses an individual adaptive learning rate $\eta_i$ for each weight . The adaptation of $\eta_i$ is determined by the sign of the partial derivation $\frac{\partial E}{\partial w_i}$ . If the sign of the derivation changes from iteration to iteration , then at least one minimum with regard to was skipped and hence the learning rate $\eta_i$ is decreased by the multiplication with a predefined constant. If there is no change of the sign, it a plateau of the objective function is assumed and the learning rate $\eta_i$ is increased by the multiplication with a second constant. Furthermore, weight-backtracking is used, i.e.. if the error increases from iteration to , then the last weight adaptation is undone. For further information about this algorithm, please refer to:

 @InCollection{silva:90b,
  author =       {Fernando M. Silva and Luis B. Almeida},
  title =        {Acceleration Techniques for the Backpropagation Algorithm},
  booktitle =    {Neural Networks -- EURASIP Workshop 1990},
  pages =        {110-119},
  year =         {1990},
  editor =       {Luis B. Almeida and C. J. Wellekens},
  number =       {412},
  series =       {LNCS},
  publisher = {Springer-Verlag},
 }

This second version of the algorithm has some minor modifications that improve the performance.

Author:: C. Igel

Date:: 1999

Changes:: none

Status:: stable

Definition at line 361 of file AdpBP.h.

Member Function Documentation

void AdpBP90b::clearMemory ( ) [inline]

Resets internal variables of the class instance.

Returns:: none

Author:: C. Igel

Date:: 1999

Changes: none

Status: stable

Definition at line 522 of file AdpBP.h.

References dedwOld, deltaw, eta, eta0, and oldError.

void AdpBP90b::init ( Model & model ) [inline, virtual]

Initializes the optimizer with default parameters.

Initializes the optimizer's parameters with the following defaults:

Increase factor for adaptive learning rate = 1.2
Decrease factor for adaptive learning rate = 0.7
Momentum factor $\alpha = 0.5$
Initial value for adaptive learning rate = 0.1

Parameters:

model

Model to be optimized.

Returns:: None.

Author:: C. Igel

Date:: 1999

Changes: none

Status: stable

Implements Optimizer.

Definition at line 390 of file AdpBP.h.

References initUserDefined().

void AdpBP90b::initUserDefined	(	Model &	model,
		double	increaseFactorP = `1.2`,
		double	decreaseFactorP = `0.7`,
		double	alphaP = `0.5`,
		double	eta0P = `0.1`
	)			`[inline]`

Initializes the optimizer with user defined parameters.

Parameters:

	model	Model to be optimized.
	increaseFactorP	Increase factor for the adaptive learning rate $\eta$ , set to 1.2 by default.
	decreaseFactorP	Decrease factor for the adaptive learning rate $\eta$ , set to 0.7 by default.
	alphaP	Momentum factor $\alpha$ that controls the effect of the momentum term, set to 0.5 by default.
	eta0P	Initial value for the adapted learning rate $\eta$ , set to 0.1 by default.

Returns:: None.

Author:: C. Igel

Date:: 1999

Changes: none

Status: stable

Definition at line 419 of file AdpBP.h.

References alpha, decreaseFactor, dedwOld, deltaw, eta, eta0, Model::getParameterDimension(), increaseFactor, oldError, and v.

Referenced by init().

double AdpBP90b::optimize	(	Model &	model,
		ErrorFunction &	errorfunction,
		const Array< double > &	input,
		const Array< double > &	target
	)			`[inline, virtual]`

Performs a run of the optimization algorithm.

The error $E^{(t)}$ and its derivatives with respect to the model parameters for the current iteration $t$ are calculated and the values of the weights $w_i,\ i = 1, \dots, n$ and the individual learning rates $\eta_i$ are adapted.

Parameters:

	model	Model to be optimized.
	errorfunction	Error function to be used for the optimization.
	input	Input vector for the model.
	target	Corresponding targets to the given inputs.