ReClaM Documentation

Reference for the Regression and Classification Models Toolbox.

Overview

ReClaM offers a rich set of methods for supervised learning, reaching from basic to state of the art tools. For example, the library provides simple linear models as well as different flavors of neural network and support vector machine models and training algorithms. Sophisticated model selection tools complete the collection.

This introduction is organized as follows:

The ReClaM Architecture

NOTE: In the transition from Shark version 1.4.x to 1.5.x there was a major change of the ReClaM architecture. For details on design goals and technical considerations refer to the following documents:

To achieve most flexibility, the ReClaM package is build like a construction kit. The main components are:

These components are modeled by the three ReClaM base classes

ReClaMBaseClasses.png

The organization of the ReClaM base classes with an illustration of the information flow.

A Typical ReClaM Application

To give a motivation and a context for the following introduction, we will very briefly outline the general structure of a typical supervised learning scenario, that is, data driven model adaptation, within the ReClaM architecture:
An instance of a problem specific Model subclass, an ErrorFunction and a suitable Optimizer object are constructed. The training data, that is, input patterns with labels, are loaded into the system. Model and optimizer are initialized. The optimization takes the form of a loop calling the optimizer. In each iteration the optimizer modifies the model parameters in order to achieve a smaller error function value than before. The error computation usually involves the model prediction on the data and the labels. This proceeding is illustrated by the following code sniplet:

 // prepare data
 Array<double> trainingData;
 Array<double> trainingTargets;
 LoadOrGenerateData(trainingData, trainingTargets);

 // prepare basic objects
 MyModel model;            // sub class of Model
 MyErrorFunction error;    // sub class of ErrorFunction
 MyOptimizer optimizer;    // sub class of Optimizer

 // optimize / learn
 optimizer.init(model);
 for (int i=0; i<ITERATIONS; i++)
 {
     optimizer.optimize(model, error, trainingData, trainingTargets);
 }

The base class interfaces provide all functionality that is needed for this general scheme. Therefore, the core of all ReClaM programs looks quite similar. Now, most of the ReClaM classes are specialized subclasses implementing models, error functions or optimizers. This document will first introduce the base class interfaces and then turn to the most prominent model types.

The Model Base Class

The Model class provides an interface for an adaptable data processing model. A typical model can be thought of as a parameterized family of functions or maps. The interface provides the following functionality:

A model needs input patterns to perform its computations. See the Model documentation for a complete list of models.

The ErrorFunction Base Class

The ErrorFunction class is a quite simple interface providing only the following functions:

An error function needs a model, input patterns and label information to perform its computations. See the ErrorFunction documentation for a complete list of error functions.

The Optimizer Base Class

The Optimizer class unifies standard initialization and iterative optimization steps of all ReClaM optimizers. It provides the following interface:

An optimizer needs a model, an error function as well as input patterns with labels to perform its computations. These computations result in a data driven adaptation of the model parameters in order to reduce the error evaluated on the training data. See the Optimizer documentation for a complete list of optimizers.

Linear Methods

ReClaM offers five simple linear models for learning in vector spaces:

The following standard learning methods operating on these models are available as optimizers:

These optimizers deviate from the standard ReClaM philosophy because they solve the underlying problems analytically, that is, without the need for an iterative update loop. In most cases, these algorithms can serve for preprocessing or as base line methods.

Neural Networks

Predefined Network Models

ReClaM comes with several predefined network types. These networks cover a wide variety of applications. Whenever these prototypes prove inappropriate they can hopefully serve as base classes or starting points for refined implementations. The following table lists several properties:

Name Type Activation Functions Training Error Measure
Hidden Neurons Output Neurons
FFNet Feed Forward
sigm1.gif
sigm1.gif
none
MSEFFNet Feed Forward
sigm1.gif
sigm1.gif
Mean Squared Error
LinOutFFNet Feed Forward
sigm1.gif
lin.gif
none
LinOutMSEBFFNet Feed Forward
sigm1.gif
lin.gif
Mean Squared Error
TanhNet Feed Forward
tanh.gif
tanh.gif
Squared Error
LinearOutputTanhNet Feed Forward
tanh.gif
lin.gif
Squared Error
ProbenNet Feed Forward
fabs.gif
lin.gif
Mean Squared Error
ProbenBNet Feed Forward
fabs.gif
lin.gif
Mean Squared Error
MSERNNet Recurrent
sigm1.gif
sigm1.gif
Mean Squared Error
RBFNet Radial Basis Function special special none
MSERBFNet Radial Basis Function special special Mean Squared Error

Connection matrix creation

The methods in the file createConnectionMatrix.h automatically create connection matrices for feed-forward and recurrent networks with several layers.
Flags are offered that can be set to establish standard connections between the neurons of the network.

Monitoring the training

The EarlyStopping class provides measures that can be used to fight overfitting during the training process.

Active Learning

By means of the VarianceEstimator class it is possible to enhance the learning speed of your network by allowing it to add a new pattern to the training set at each step. This pattern is chosen in a way that it will minimize the network error.

Kernel Methods

ReClaM offers a variety of kernel based learning algorithms. In the following the basic concepts and the corresponding classes are introduced.

svm.png

Kernel Functions

Kernel based learning algorithms rely on a positive definite kernel function taking two input patterns as arguments. The kernel function can be interpreted as first mapping the inputs into a (usually high dimensional) Hilbert space and then computing the inner product in this space. A kernel based algorithm is a learning algorithm that can be formulated in terms of inner products of pairs of input patterns. These inner products are then computed using a kernel function.

The KernelFunction interface encapsulates this functionality. Usually a parameterized family of kernel functions is considered. Because we are interested in the optimization of its parameters, KernelFunction is derived from Model. A kernel does not provide an output given an input. Therefore it defines a new interface:

Of course the KernelFunction inherits the Model members for retrieving and setting parameters and checking the feasibility of the parameters.

Several standard kernel functions are available, together with combinations of kernels to new kernels:

Class Parameters Formula Description
LinearKernel none $ \langle x, z \rangle = x^T z $ The linear kernel is the canonical inner product.
PolynomialKernel degree d and offset v $ (\langle x, z \rangle + v)^d $ The polynomial kernel is a polynomial of the linear kernel. This representation corresponds to a feature space of monomials of the input coordinates.
RBFKernel concentration $ \gamma $ $ \exp(-\gamma \|x-z\|^2) $ The Gaussian Radial Basis Function kernel is one of the most widely used kernels for vector valued data. It is known to be universal.
DiagGaussKernel diagonal inverse covariance matrix M $ \exp(-(x-z)^T M (x-z)) $ Gaussian kernel with adaptible covariance matrix. The extent of the Gaussian along each coordinate axis can be controlled.
GeneralGaussKernel inverse covariance matrix M $ \exp(-(x-z)^T M (x-z)) $ Gaussian kernel with adaptible covariance matrix. The complete shape of the Gaussian can be controlled.
NormalizedKernel parameters of a base kernel k $ \frac{k(x, z)}{\sqrt{k(x, x) \cdot k(z, z)}} $ The normalization of a positive definite kernel is again a valid positive definite kernel. The normalization ensures that the norm of each example in feature space is 1.
WeightedSumKernel non-negative weights wi and parameters of base kernels ki $ \sum_i w_i k_i(x, z) $ The non-negative linear combination of positive definite kernels is a valid positive definite kernel again.

Simple Algorithms

The most simple kernel based algorithms available are

Their standard vector space versions can be obtained by simply using the linear kernel. These algorithms can serve as a baseline for the evaluation of more advanced methods. A special aspect of these algorithms is that they do not have any hyperparameters and do not need any special optimization. That is, once the training data are provided the models can be used immediately.

Support Vector Machines

Support Vector Machines (SVMs) are the most prominent kernel based learning algorithms. The basic ideas of support vector machines are as follows:

SVM Model

Support vector machines usually result in a sparse affine sum of the form

\[ f(x) = \sum_{i \in S} \alpha_i k(x_i, x) + b \]

where the support vector index set S is a subset of the training data indices. In the case of classification with more than two classes the expansion becomes

\[ f(x) = \sum_{i \in S} \sum_c \alpha_{i,c} k(x_i, x) y_c + b \]

where c sums over the classes and $ y_i $ are class prototype vectors.

In ReClaM, a support vector machine is a Model holding the coefficients $ \alpha_i $ and $ b $. Its prediction can be restricted to the sign of the computed value, which is the standard form for binary classification problems. For multi-class problems, the output vector can analogously be transformed into an integer class label. In contrast to other ReClaM models, support vector machines output only one value per input pattern. They can be viewed as affine linear functions mapping from the span of the support vectors in feature space to the real line (or to $R^n$).

SVM Training

Training an SVM involves several so called hyperparameters. These are encapsulated by the MetaSVM class and its descendants:

Class Parameters Problem Type Description
C_SVM regularization parameter C binary classification C-SVM formulation using either 1-norm or 2-norm slack penalty. This is the most usual way to train a support vector machine.
Epsilon_SVM regularization parameter C, accuracy parameter $ \varepsilon $ regression Standard SVM for regression
RegularizationNetwork regularization parameter $ \gamma $ regression regularized linear regression in feature space
GaussianProcess none regression Bayesian inference is used to estimate the regularization and kernel parameters.
OneClassSVM regularization parameter $ \nu $ support / density quantile estimation Estimation of an area of high density
AllInOneMcSVM regularization parameter C multi-category classification Covers the methods by Weston and Watkins and by Wahba (without bias)
CrammerSingerMcSVM regularization parameter $ \beta $ multi-category classification Multi-Class SVM by Crammer and Singer (without bias)
OVAMcSVM regularization parameter C multi-category classification A simple one-versus-all machine (without bias)
BinaryCostMcSVM regularization parameter C multi-category classification Multi-class classification at the training cost of a binary machine (without bias)

For the adaptation of the hyperparameters of the C_SVM or the Epsilon_SVM, please refer to the model selection section below.

SVMs are trained solving a special quadratic program. For this purpose ReClaM comes with a various efficient solvers for the particular types of optimization problems. The quadratic programming is hidden from the user of the library. An SVM is trained using the special SVM_Optimizer class. This class constructs a QuadraticProgram object, solves the corresponding problem and copies the solution into the SVM parameter vector. The optimization involves a special quadratic objective function. For efficiency reasons it is not desirable to define this objective function as an ErrorFunction derived class and use a standard purpose gradient based optimizer. Because of this special situation the SVM_Optimizer can be called without an error function parameter. When called with an error function through the standard Optimizer interface the error function is ignored.

Model Selection

The process of model selection is the choice of a model from a family of candidate models. This may correspond to the selection of the hyperparameters of a MetaSVM subclass. In this case the model family is parameterized by the MetaSVM parameter vector.

General Model Selection

A model selection problem is usually solved by an outer optimization loop investigating a model family and an inner optimization loop finding the best model within this family. The fitness of a model family is simply the fitness of its best element.

In general, model selection can be carried out on complicated spaces like the possible topologies of a neural network. These problems require specialized evolutionary algorithms not provided by the ReClaM library. However, in the case of support vector machines the model selection problem can usually be reduced to real valued parameter adaptation, to which ReClaM is perfectly suited. Therefore most model selection support is available for this model type. Only the cross validation method is applicable to general model selection problems.

Model Selection for Support Vector Machines

The model of a support vector machine is made up by the kernel function and the complexity control. The kernel parameters and the SVM training parameters are to be chosen. These parameters are captured by the MetaSVM class and its subclasses.

Several objective functions have been proposed for model selection for support vector machines. Besides the usual cross validation measures ReClaM provides the following ErrorFunction subclasses:

Example Programs

To give you a better idea, how using the several components of ReClaM to build your own networks, you can take a look at some commented source codes of example programs for different types of predefined networks. To run these programs please go to the examples directory of ReClaM.

Please have a look at our introductory tutorials on using multi-layer perceptron networks, part 1 and part 2.

Models, Error Functions, and Optimizers

Models

The following table lists the most prominent ReClaM model types:

Name Inputs Outputs Description
LinearMap $ R^n $ $ R^m $ Linear mapping $ x \mapsto y = A x $. An affine version is available, too.
LinearFunction $ R^n $ $ R $ Special case of a linear map with one output dimension. An affine version is available, too.
ComponentWiseModel $ X^k $ $ Y^k $ Apply a base model $ f : X \rightarrow Y $ $k$-fold and component wise, making up a model $ F : X^k \rightarrow Y^k $.
ConcatenatedModel $ X $ $ Y $ Apply a chain $ f_1 \circ f_2 \circ \dots \circ f_k $ of models. The output space of model $ f_i $ and the input space of model $ f_{i+1} $ must match.
FFNet $ R^n $ $ R^m $ feed forward neural network
RBFNet $ R^n $ $ R $ radial basis function network
MSERNNet $ R^n $ $ R^m $ recurrent neural network with built-in mean squared error computation
KernelFunction $ X \times X $ $ R $ Please refer to the Kernel Methods documentation.
KernelMeanClassifier $ X $ $ R $ Simple mean classifier in a kernel defined feature space
KernelNearestNeighbor $ X $ $ \{-1, +1\} $ Simple k-nearest-neighbor classifier in a kernel defined feature space
SVM $ X $ $ \{-1, +1\} $ or $ R $ Support Vector Machine model, that is, affine linear function in a kernel defined feature space.
MultiClassSVM $ X $ $ \{0, \dots, \#classes-1\} $ or $ R^n $ Support Vector Machine model, that is, affine linear function in a kernel defined feature space.
SvmApproximationModel Used internally. Please refer to the SvmApproximation class for documentation.
MetaSVM --- --- Base class of all SVM training schemes. See the MetaSVM documentation or the section Support Vector Machines for details.
SigmoidModel R (0, 1) sigmoidal function $ f(x) = \frac{1}{1 + \exp(ax+b)} $




Error Functions

This table gives an overview over the error functions available:

Class Differentiable Restrictions Description
SquaredError yes none squared error between model output and target
MeanSquaredError / DF_MeanSquaredError yes none mean squared error between model output and target
CrossEntropy / DF_CrossEntropy yes unit interval model outputs cross entropy measure for neural network classifier training
CrossEntropyIndependent / DF_CrossEntropyIndependent yes unit interval model outputs cross entropy measure for neural network classifier training with non exclusive attributes
ClassificationError no Classifiers fraction of misclassified patterns
BalancedClassificationError no Classifiers mean fraction of misclassified patterns per class
ErrorPercentage no none error percentage based on the mean squared error
RadiusMargin yes KernelFunction derived classes and C_SVM with 2-norm slack penalty squared radius margin quotient for SVM model selection
NegativeKTA yes KernelFunction derived classes and C_SVM with 2-norm slack penalty negative kernel target alignment
NegativeBKTA yes KernelFunction derived classes and C_SVM with 2-norm slack penalty negative balanced kernel target alignment
NegativePolarization yes KernelFunction derived classes and C_SVM with 2-norm slack penalty negative kernel polarization measure
SpanBound no SVM classifier span bound for a support vector machine classifier model selection
LOO no SVM classifier leave one out error especially tuned towards support vector machines
InverseClassSeparability yes KernelFunction derived classes inverse of the class separability measure termed J, by Huilin Xiong and M. N. S. Swamy




Optimizers

Among others, here come the most used optimizers:

Class Parameters Gradient-Based? Description
IRpropPlus initial step sizes, increase and decrease factors, minimal and maximal step sizes yes Improved resilent backpropagation with weight backtracking. Other variants without improvement or backtracking are available.
CG line search type and several tuning parameters with default values yes Conjugate Gradient method
BFGS line search type and several tuning parameters with default values yes Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm
CMAOptimizer CMA type, initial standard deviations no Covariance Matrix Adaptation Evolution Strategy. This is a wrapper class for the EALib CMA implementation. The CMA is the first choice optimizer if the derivative is not available or lots of undesirable local minima are a concern.
GridSearch grid definition no Grid search is the most basic optimization method available. Several variants add flexibility and/or improve efficiency.

Further features

ReClaM offers a bunch of other models, error functions, and optimizers, as well as tools, e.g. for data handling, unified exception handling, and even more. Check out the class or file lists for a full overview.

For example, have a look at the tools defined in

Maybe the fastest way to get a grip on ReClaM is to have a look at the examples.