EarlyStopping Class Reference

Used for monitoring purposes, to avoid overfitting. More...

#include <EarlyStopping.h>

List of all members.

Public Member Functions

 EarlyStopping (unsigned)
 A new instance of EarlyStopping is created, internal variables are initialized.
double getGL ()
 Returns the current value of the generalization loss.
double getTP ()
 Returns the current value of the training progress.
double getPQ ()
 Returns the current value of the performance quotient.
unsigned getUP ()
 Returns the current number of generalization loss increases.
bool GL (double)
 Checks, whether the generalization loss exceeds a certain value.
bool GL ()
 Checks, whether the generalization loss exceeds "1". value.
bool TP (double)
 Checks, whether the trainings progress falls short of a certain value.
bool TP ()
 Checks, whether the trainings progress falls short of "0.1".
bool PQ (double)
 Checks, whether the performance quotient exceeds a certain value.
bool PQ ()
 Checks, whether the performance quotient exceeds "0.5".
bool UP (unsigned)
 Checks, whether the number of generalization loss increases is equal to or greater than a certain value.
bool UP ()
 Checks, whether the number of generalization loss increases is equal to or greater than "3".
bool one_of_all (double, double, double, unsigned)
 Checks, whether one of the performance characteristics is out of range.
bool one_of_all ()
 Checks, whether one of the performance characteristics is out of default range.
void update (double, double)
 The performance characteristics are evaluated for the current time step.

Detailed Description

Used for monitoring purposes, to avoid overfitting.

When training neural networks, there is always the danger of overfitting, i.e. the prediction error for a training set is monotonically decreasing with the number of optimization algorithm iterations, but the error for a validation set (i.e. independent data) is first decreasing, followed by an increase. To achieve best generalization performance, the training should then be stopped at an iteration step when the validation set error has its minimum.
This class offers methods, that can be used to monitor the performance of a network related to a couple of performance characteristics, so one can decide when to stop the training.
The given performance characteristics are:

The training progress (TP) given as $1000 \ast \frac{E_{T_{average}}}{E_{T_{min}} - 1}$, where $E_{T_{average}}$ is the average training set error evaluated over a period of timesteps (striplength) and $E_{T_{min}}$ is the minimal error for the training set during this period.

The generalization loss (GL) given as $100 \ast \frac{E_{V}}{E_{V_{min}} - 1}$, where $E_{V}$ is the current error of the validation set and $E_{V_{min}}$ is the minimum error of the validation set, as calculated during the striplength.

The number of generalization loss increases (UP) is given as the number of time steps during the striplength, at which the generalization loss increases compared to the previous time step.

The performance quotient (PQ) given as $\frac{GL}{TP}$.

Example
  void main()
  {

      // Create net:
      MyNet net;

      // Fill net:
      ...

      // Create early stopping instance for monitoring 5 time steps:
      EarlyStopping estop( 5 );

      // Training the net:
      for ( unsigned epoch = 1; epoch < max_epochs; epoch++ )
      {
          // Train net with prefered optimization algorithm:
          ...

          // Calculate training and validation error of
          // current model:
          ...

          // Update of monitoring variables:
          estop.update( error_train, error_validation );

          // Overfitting of network? Then stop training:
          if ( estop.one_of_all( 1.0, 1.0, 1.0, 3) ) break;

      } // next training epoch
  }
Author:
M. Hüsken
Date:
1999
Changes:
none
Status:
stable

Definition at line 142 of file EarlyStopping.h.

Constructor & Destructor Documentation

EarlyStopping::EarlyStopping ( unsigned  sl = 5  ) 

A new instance of EarlyStopping is created, internal variables are initialized.

A new instance of EarlyStopping is created, internal variables are initialized.

The internal variables of the new instance are initialized and the time interval used for monitoring is set.

Parameters:
sl The number of time steps (striplength), for which the performance shall be evaluated. The default values is "5".
Returns:
none
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 81 of file EarlyStopping.cpp.

Member Function Documentation

double EarlyStopping::getGL (  ) 

Returns the current value of the generalization loss.

Returns:
The current generalization loss.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 175 of file EarlyStopping.cpp.

double EarlyStopping::getPQ (  ) 

Returns the current value of the performance quotient.

Call this method only if the current time step equals or exceeds the predefined striplength. Because of the calculation of the performance quotient you will get a value without significance otherwise.

Returns:
The current performance quotient.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 231 of file EarlyStopping.cpp.

double EarlyStopping::getTP (  ) 

Returns the current value of the training progress.

Call this method only if the current time step equals or exceeds the predefined striplength. Because of the calculation of the training progress you will get a value without significance otherwise.

Returns:
The current generalization loss.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 202 of file EarlyStopping.cpp.

unsigned EarlyStopping::getUP (  ) 

Returns the current number of generalization loss increases.

Returns:
The current number of generalization loss increases.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 256 of file EarlyStopping.cpp.

bool EarlyStopping::GL (  ) 

Checks, whether the generalization loss exceeds "1". value.

Returns:
"true", if the generalization loss exceeds "1", "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Referenced by one_of_all().

bool EarlyStopping::GL ( double  alpha = 1.0  ) 

Checks, whether the generalization loss exceeds a certain value.

Checks, whether the generalization loss exceeds a certain value.

Parameters:
alpha The value that is compared to the generalization loss. The default value is "1".
Returns:
"true", if the generalization loss exceeds alpha, "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 282 of file EarlyStopping.cpp.

bool EarlyStopping::one_of_all (  ) 

Checks, whether one of the performance characteristics is out of default range.

This method will check all performance characteristics:

  • The generalization loss is out of range, when it is greater than "1"
  • The trainings progress is out of range, when it is less than "0.1"
  • The performance quotient is out of range, when it is greater than "0.5"
  • The number of generalization loss increases is out of range, when it is equal to or greater than "3"

If at least one of the performance characteristics is out of range the method will return "true".

Returns:
"true", if at least one of the performance characteristics is out of range, "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

bool EarlyStopping::one_of_all ( double  alpha1 = 1.0,
double  alpha2 = 1.0,
double  alpha3 = 1.0,
unsigned  s = 3 
)

Checks, whether one of the performance characteristics is out of range.

Checks, whether one of the performance characteristics is out of range.

This method will check all performance characteristics against the given parameters:

  • The generalization loss is out of range, when it is greater than alpha1
  • The trainings progress is out of range, when it is less than alpha2
  • The performance quotient is out of range, when it is greater than alpha3
  • The number of generalization loss increases is out of range, when it is equal to or greater than s

If at least one of the performance characteristics is out of range the method will return "true".

Parameters:
alpha1 The value that is compared to the generalization loss. The default value is "1".
alpha2 The value that is compared to the trainings progress The default value is "0.1".
alpha3 The value that is compared to the performance quotient The default value is "0.5".
s The value that is compared to the number of generalization loss increases. The default value is "3".
Returns:
"true", if at least one of the performance characteristics is out of range, "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 424 of file EarlyStopping.cpp.

References GL(), PQ(), TP(), and UP().

bool EarlyStopping::PQ (  ) 

Checks, whether the performance quotient exceeds "0.5".

Call this method only if the current time step equals or exceeds the predefined striplength. Because of the calculation of the performance quotient you will get a value without significance otherwise.

Returns:
"true", if the performance quotient exceeds "0.5", "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Referenced by one_of_all().

bool EarlyStopping::PQ ( double  alpha = 0.5  ) 

Checks, whether the performance quotient exceeds a certain value.

Checks, whether the performance quotient exceeds a certain value.

Call this method only if the current time step equals or exceeds the predefined striplength. Because of the calculation of the performance quotient you will get a value without significance otherwise.

Parameters:
alpha The value that is compared to the performance quotient The default value is "0.5".
Returns:
"true", if the performance progress exceeds alpha, "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 346 of file EarlyStopping.cpp.

bool EarlyStopping::TP (  ) 

Checks, whether the trainings progress falls short of "0.1".

Call this method only if the current time step equals or exceeds the predefined striplength. Because of the calculation of the trainings progress you will get a value without significance otherwise.

Returns:
"true", if the trainings progress falls short of "0.1", "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Referenced by one_of_all().

bool EarlyStopping::TP ( double  alpha = 0.1  ) 

Checks, whether the trainings progress falls short of a certain value.

Checks, whether the trainings progress falls short of a certain value.

Call this method only if the current time step equals or exceeds the predefined striplength. Because of the calculation of the trainings progress you will get a value without significance otherwise.

Parameters:
alpha The value that is compared to the trainings progress The default value is "0.1".
Returns:
"true", if the trainings progress falls short of alpha, "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 313 of file EarlyStopping.cpp.

bool EarlyStopping::UP (  ) 

Checks, whether the number of generalization loss increases is equal to or greater than "3".

Returns:
"true", if the number of generalization loss increases is equal to or greater than "3", "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Referenced by one_of_all().

bool EarlyStopping::UP ( unsigned  s = 3  ) 

Checks, whether the number of generalization loss increases is equal to or greater than a certain value.

Checks, whether the number of generalization loss increases is equal to or greater than a certain value.

Parameters:
s The value that is compared to the number of generalization loss increases. The default value is "3".
Returns:
"true", if the number of generalization loss increases is equal to or greater than alpha, "false" otherwise.
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 375 of file EarlyStopping.cpp.

void EarlyStopping::update ( double  e,
double  e_va 
)

The performance characteristics are evaluated for the current time step.

The performance characteristics are evaluated for the current time step.

Given the errors for the training and validation set for the current time step, the performance characteristics Generalization Loss, Trainings Progress and number of generalization loss increases are evaluated. When the current time step (= no. of method calls) is equal to the predefined striplength, the values are reset.

Parameters:
e The current training set error.
e_va The current validation set error.
Returns:
none
Author:
M. Hüsken
Date:
1999
Changes
none
Status
stable

Definition at line 117 of file EarlyStopping.cpp.

The documentation for this class was generated from the following files: