mlpack

đź”— RandomForest

The RandomForest class implements a parallelized random forest classifier that supports numerical and categorical features, by default using Gini gain to choose which feature to split on in each tree.

Random forests are a collection of decision trees that give better performance than a single decision tree. They are useful for classifying points with discrete labels (i.e. 0, 1, 2). This implementation of the RandomForest class is not for regression (i.e. predicting continuous values).

mlpack’s RandomForest class offers configurability via template parameters and runtime parameters. This is used to provide the additional API-compatible ExtraTrees class. To use ExtraTrees, simply replace RandomForest with ExtraTrees in any of the documentation below. (More information…)

Simple usage example:

// Train a random forest on random numeric data and predict labels on test data:

// All data and labels are uniform random; 10 dimensional data, 5 classes.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));
arma::mat testData(10, 500, arma::fill::randu); // 500 test points.

mlpack::RandomForest rf;            // Step 1: create model.
rf.Train(dataset, labels, 5, 10);   // Step 2: train model.
arma::Row<size_t> predictions;
rf.Classify(testData, predictions); // Step 3: classify points.
// You can also use `ExtraTrees` instead of `RandomForest`!

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 3) << " test points classified as class "
    << "3." << std::endl;

More examples...

See also:

đź”— Constructors

Constructor Parameters:

name type description default
data arma::mat Column-major training matrix. (N/A)
info data::DatasetInfo Dataset information, specifying type information for each dimension. (N/A)
labels arma::Row<size_t> Training labels, between 0 and numClasses - 1 (inclusive). Should have length data.n_cols. (N/A)
numClasses size_t Number of classes in the dataset. (N/A)
weights arma::rowvec Instance weights for each training point. Should have length data.n_cols. (N/A)
numTrees size_t Number of trees to train in the random forest. 20
       
minLeafSize size_t Minimum number of points in each leaf node of each decision tree. 1
minGainSplit double Minimum gain for a node to split in each decision tree. 1e-7
maxDepth size_t Maximum depth for each decision tree. (0 means no limit.) 0
warmStart bool (Only available in Train().) If true, training adds numTrees trees to the random forest. If false, an entirely new random forest will be created. false

Note: different types can be used for data and weights (e.g., arma::fmat, arma::sp_mat). However, the element type of data and weights must match; for example, if data has type arma::fmat, then weights must have type arma::frowvec.

đź”— Training

If training is not done as part of the constructor call, it can be done with one of the following versions of the Train() member function:

Types of each argument are the same as in the table for constructors above.

Notes:

đź”— Classification

Once a RandomForest is trained, the Classify() member function can be used to make class predictions for new data.

Classification Parameters:

usage name type description
single-point point arma::vec Single point for classification.
single-point prediction size_t& size_t to store class prediction into.
single-point probabilitiesVec arma::vec& arma::vec& to store class probabilities into. Will be set to length numClasses.
       
multi-point data arma::mat Set of column-major points for classification.
multi-point predictions arma::Row<size_t>& Vector of size_ts to store class prediction into. Will be set to length data.n_cols.
multi-point probabilities arma::mat& Matrix to store class probabilities into (number of rows will be equal to number of classes, number of columns will be equal to data.n_cols).

Note: different types can be used for data and point (e.g. arma::fmat, arma::sp_mat, arma::sp_vec, etc.). However, the element type that is used should be the same type that was used for training.

đź”— Other Functionality

For complete functionality, the source code can be consulted. Each method is fully documented.

đź”— Simple Examples

See also the simple usage example for a trivial use of RandomForest.

Train a random forest incrementally on random mixed categorical data and save it to disk:

// Load a categorical dataset.
arma::mat dataset;
mlpack::data::DatasetInfo info;
// See https://datasets.mlpack.org/covertype.train.arff.
mlpack::data::Load("covertype.train.arff", dataset, info, true);

arma::Row<size_t> labels;
// See https://datasets.mlpack.org/covertype.train.labels.csv.
mlpack::data::Load("covertype.train.labels.csv", labels, true);

// Create the random forest.
mlpack::RandomForest rf;
// Train 10 trees on the given dataset, with a minimum leaf size of 3.
rf.Train(dataset, info, labels, 7 /* classes */, 10 /* trees */,
         3 /* minimum leaf size */);

// Now load categorical test data.
arma::mat testDataset;
// See https://datasets.mlpack.org/covertype.test.arff.
mlpack::data::Load("covertype.test.arff", testDataset, info, true);

arma::Row<size_t> testLabels;
// See https://datasets.mlpack.org/covertype.test.labels.csv.
mlpack::data::Load("covertype.test.labels.csv", testLabels, true);

// Compute test set accuracy.
arma::Row<size_t> testPredictions;
rf.Classify(testDataset, testPredictions);
double accuracy = 100.0 * ((double) arma::accu(testPredictions == testLabels)) /
    testLabels.n_elem;
std::cout << "After training 10 trees, test set accuracy is " << accuracy
    << "%." << std::endl;

// Now train another 10 trees and compute the test accuracy.
rf.Train(dataset, info, labels, 7 /* classes */, 10 /* trees */,
         3 /* minimum leaf size */, 0.0 /* minimum split gain */,
         0 /* maximum depth (unlimited) */, true /* incremental training */);

rf.Classify(testDataset, testPredictions);
accuracy = 100.0 * ((double) arma::accu(testPredictions == testLabels)) /
    testLabels.n_elem;
std::cout << "After training 20 trees, test set accuracy is " << accuracy
    << "%." << std::endl;

// Save the random forest to disk.
mlpack::data::Save("rf.bin", "rf", rf);

Load a random forest and print some information about it.

mlpack::RandomForest rf;
// This call assumes a random forest called "rf" has already been saved to
// `rf.bin` with `data::Save()`.
mlpack::data::Load("rf.bin", "rf", rf, true);

std::cout << "The random forest in 'rf.bin' contains " << rf.NumTrees()
    << " trees." << std::endl;
if (rf.NumTrees() > 0)
{
  std::cout << "The first tree's root node has " << rf.Tree(0).NumChildren()
      << " children." << std::endl;
}

Train a random forest on categorical data, and compare its performance with the performance of each individual tree:

// Load a categorical dataset (training and test sets).
arma::mat dataset, testDataset;
mlpack::data::DatasetInfo info;
arma::Row<size_t> labels, testLabels;

// See the following files:
//  * https://datasets.mlpack.org/covertype.train.arff
//  * https://datasets.mlpack.org/covertype.train.labels.csv
//  * https://datasets.mlpack.org/covertype.test.arff
//  * https://datasets.mlpack.org/covertype.test.labels.csv
mlpack::data::Load("covertype.train.arff", dataset, info, true);
mlpack::data::Load("covertype.train.labels.csv", labels, true);
mlpack::data::Load("covertype.test.arff", testDataset, info, true);
mlpack::data::Load("covertype.test.labels.csv", testLabels, true);

// Create the random forest.
mlpack::RandomForest rf;
// Train 20 trees on the given dataset, with a minimum leaf size of 5.
rf.Train(dataset, info, labels, 7 /* classes */, 20 /* trees */,
         5 /* minimum leaf size */);

// Compute test set accuracy for each tree.
arma::Row<size_t> testPredictions;
for (size_t i = 0; i < rf.NumTrees(); ++i)
{
  rf.Tree(i).Classify(testDataset, testPredictions);
  const double accuracy = 100.0 *
      ((double) arma::accu(testPredictions == testLabels)) / testLabels.n_elem;
  std::cout << "Tree " << i << " has test accuracy " << accuracy << "%."
      << std::endl;
}

// Now compute accuracy using the whole forest.
rf.Classify(testDataset, testPredictions);
const double accuracy = 100.0 *
    ((double) arma::accu(testPredictions == testLabels)) / testLabels.n_elem;
std::cout << "The whole forest has test accuracy " << accuracy << "%."
    << std::endl;

Train an ExtraTrees model on random numeric data.

// 1000 random points in 10 dimensions.
arma::mat dataset(10, 1000, arma::fill::randu);
// Random labels for each point, totaling 5 classes.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));

// Train in the constructor, using 10 trees in the forest.
// Note that `ExtraTrees` has exactly the same API as `RandomForest`.
mlpack::ExtraTrees<> rf(dataset, labels, 5, 10);

// Create a single test point.
arma::vec testPoint(10, arma::fill::randu);

size_t prediction;
arma::vec probabilities;
rf.Classify(testPoint, prediction, probabilities);
std::cout << "Test point predicted to be class " << prediction << "."
    << std::endl;
std::cout << "Probabilities of each class: " << probabilities.t();

See also the following fully-working examples:

đź”— Advanced Functionality: Template Parameters

Using different element types.

RandomForest’s constructors, Train(), and Predict() functions support any data type, so long as it supports the Armadillo matrix API. So, for instance, learning can be done on single-precision floating-point data:

// 1000 random points in 10 dimensions.
arma::fmat dataset(10, 1000, arma::fill::randu);
// Random labels for each point, totaling 5 classes.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));

// Train in the constructor.
mlpack::RandomForest rf(dataset, labels, 5);

// Create test data (500 points).
arma::fmat testDataset(10, 500, arma::fill::randu);
arma::Row<size_t> predictions;
rf.Classify(testDataset, predictions);
// Now `predictions` holds predictions for the test dataset.

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 0) << " test points classified as class "
    << "0." << std::endl;

Fully custom behavior.

mlpack provides a few variants of the random forest classifier, using the template parameters of the RandomForest class. The following types can be used as drop-in replacements throughout this documentation page:

Fully custom classes can also be used to control the behavior of the RandomForest class. The full signature of the class is as follows:

RandomForest<FitnessFunction,
             DimensionSelectionType,
             NumericSplitType,
             CategoricalSplitType,
             UseBootstrap,
             BootstrapType>

An additional RandomForest constructor offers two additional parameters to pass the DimensionSelectionType, via the dimSelector argument, and the BootstrapType, via the bootstrap argument, in case they have non-default constructors. See the BootstrapType documentation.

Note that the first four of these template parameters are exactly the same as the template parameters for the DecisionTree class.

Below, details are given for the requirements of each of these template types.

FitnessFunction

// You can use this as a starting point for implementation.
class CustomFitnessFunction
{
  // Return the range (difference between maximum and minimum gain values).
  double Range(const size_t numClasses);

  // Compute the gain for the given vector of labels, where `labels[i]` has an
  // associated instance weight `weights[i]`.
  //
  // `RowType` and `WeightVecType` will be vector types following the Armadillo
  // API.  If `UseWeights` is `false`, then the `weights` vector should be
  // ignored (e.g. the labels are not weighted).
  template<bool UseWeights, typename RowType, typename WeightVecType>
  double Evaluate(const RowType& labels,
                  const size_t numClasses,
                  const WeightVecType& weights);

  // Compute the gain for the given counted set of labels, where `counts[i]`
  // contains the number of points with label `i`.  There are `totalCount`
  // labels total, and `counts` has length `numClasses`.
  //
  // `UseWeights` is ignored, and `CountType` will be an integral type (e.g.
  // `size_t`).
  template<bool UseWeights, typename CountType>
  double EvaluatePtr(const CountType* counts,
                     const size_t numClasses,
                     const CountType totalCount);
};

DimensionSelectionType

class CustomDimensionSelect
{
 public:
  // Get the first dimension to try.
  // This should return a value between `0` and `data.n_rows`.
  size_t Begin();

  // Get the next dimension to try.  Note that internal state can be used to
  // track which candidate dimension is currently being looked at.
  // This should return a value between `0` and `data.n_rows`.
  size_t Next();

  // Get a value indicating that all dimensions have been tried.
  size_t End() const;

  // The usage pattern of `DimensionSelectionType` by `DecisionTree` is as
  // follows, assuming that `dim` is an instantiated `DimensionSelectionType`
  // object:
  //
  // for (size_t dim = dim.Begin(); dim != dim.End(); dim = dim.Next())
  // {
  //   // ... try to split on dimension `dim` ...
  // }
};

NumericSplitType

template<typename FitnessFunction>
class CustomNumericSplit
{
 public:
  // This class can hold any extra data that is necessary to encode a split.  It
  // should only be non-empty if a single `double` value cannot be used to hold
  // the information corresponding to a split.
  class AuxiliarySplitInfo { };

  // If a split with better resulting gain than `bestGain` is found, then
  // information about the new, better split should be stored in `splitInfo` and
  // `aux`.  Specifically, a split is better than `bestGain` if the sum of the
  // gains that the children will have (call this `sumChildrenGains`) is
  // sufficiently better than the gain of the unsplit node (call this
  // `unsplitGain`):
  //
  //    split if `sumChildrenGains - unsplitGain > bestGain`, and
  //             `sumChildrenGains - unsplitGain > minGainSplit`, and
  //             each child will have at least `minLeafSize` points
  //
  // The new best split value should be returned (or anything greater than or
  // equal to `bestGain` if no better split is found).
  //
  // If a new best split is found, then `splitInfo` and `aux` should be
  // populated with the information that will be needed for
  // `CalculateDirection()` to successfully choose the child for a given point.
  // `splitInfo` should be set to a vector of length 1.  The format of `aux` is
  // arbitrary and is detailed more below.
  //
  // If `UseWeights` is false, the vector `weights` should be ignored.
  // Otherwise, they are instance weighs for each value in `data` (one dimension
  // of the input data).
  template<bool UseWeights, typename VecType, typename WeightVecType>
  static double SplitIfBetter(const double bestGain,
                              const VecType& data,
                              const arma::Row<size_t>& labels,
                              const size_t numClasses,
                              const WeightVecType& weights,
                              const size_t minLeafSize,
                              const double minGainSplit,
                              arma::vec& splitInfo,
                              AuxiliarySplitInfo& aux);

  // Return the number of children for a given split (stored as the single
  // element from `splitInfo` and auxiliary data `aux` in `SplitIfBetter()`).
  static size_t NumChildren(const arma::vec& splitInfo,
                            const AuxiliarySplitInfo& aux);

  // Given a point with value `point`, and split information `splitInfo` and
  // `aux`, return the index of the child that corresponds to the point.  So,
  // e.g., if the split type was a binary split on the value `splitInfo`, you
  // might return `0` if `point < splitInfo`, and `1` otherwise.
  template<typename ElemType>
  static size_t CalculateDirection(
      const ElemType& point,
      const arma::vec& splitInfo,
      const AuxiliarySplitInfo& /* aux */);
};

CategoricalSplitType

template<typename FitnessFunction>
class CustomCategoricalSplit
{
 public:
  // This class can hold any extra data that is necessary to encode a split.  It
  // should only be non-empty if a single `double` value cannot be used to hold
  // the information corresponding to a split.
  class AuxiliarySplitInfo { };

  // If a split with better resulting gain than `bestGain` is found, then
  // information about the new, better split should be stored in `splitInfo` and
  // `aux`.  Specifically, a split is better than `bestGain` if the sum of the
  // gains that the children will have (call this `sumChildrenGains`) is
  // sufficiently better than the gain of the unsplit node (call this
  // `unsplitGain`):
  //
  //    split if `sumChildrenGains - unsplitGain > bestGain`, and
  //             `sumChildrenGains - unsplitGain > minGainSplit`, and
  //             each child will have at least `minLeafSize` points
  //
  // The new best split value should be returned (or anything greater than or
  // equal to `bestGain` if no better split is found).
  //
  // If a new best split is found, then `splitInfo` and `aux` should be
  // populated with the information that will be needed for
  // `CalculateDirection()` to successfully choose the child for a given point.
  // `splitInfo` should be set to a vector of length 1.  The format of `aux` is
  // arbitrary and is detailed more below.
  //
  // If `UseWeights` is false, the vector `weights` should be ignored.
  // Otherwise, they are instance weighs for each value in `data` (one
  // categorical dimension of the input data, which takes values between `0` and
  // `numCategories - 1`).
  template<bool UseWeights, typename VecType, typename LabelsType,
           typename WeightVecType>
  static double SplitIfBetter(
      const double bestGain,
      const VecType& data,
      const size_t numCategories,
      const LabelsType& labels,
      const size_t numClasses,
      const WeightVecType& weights,
      const size_t minLeafSize,
      const double minGainSplit,
      arma::vec& splitInfo,
      AuxiliarySplitInfo& aux);

  // Return the number of children for a given split (stored as the single
  // element from `splitInfo` and auxiliary data `aux` in `SplitIfBetter()`).
  static size_t NumChildren(const arma::vec& splitInfo,
                            const AuxiliarySplitInfo& aux);

  // Given a point with (categorical) value `point`, and split information
  // `splitInfo` and `aux`, return the index of the child that corresponds to
  // the point.  So, e.g., for `AllCategoricalSplit`, which splits a categorical
  // dimension into one child for each category, this simply returns `point`.
  template<typename ElemType>
  static size_t CalculateDirection(
      const ElemType& point,
      const arma::vec& splitInfo,
      const AuxiliarySplitInfo& /* aux */);
};

UseBootstrap

Note: this parameter will be removed in mlpack 5.0.0. A value of false will then be equivalent to setting BootstrapType to IdentityBootstrap, and a value of true will be equivalent to DefaultBootstrap.

BootstrapType

class CustomBootstrapType
{
 public:
  /**
   * Compute a bootstrap dataset based on the original dataset.
   * If `UseWeights` is `false`, then `weights` and `bootstrapWeights` can be
   * ignored.
   *
   * When the function is complete, `bootstrapDataset` and `bootstrapLabels`
   * should contain a bootstrapped dataset.  If `UseWeights` is `true`, then
   * `bootstrapWeights` should contain the corresponding instance weights for
   * the bootstrapped dataset.
   */
  template<
      bool UseWeights,
      typename MatType,
      typename LabelsType,
      typename WeightsType>
  void Bootstrap(
      const MatType& dataset,
      const LabelsType& labels,
      const WeightsType& weights,
      MatType& bootstrapDataset,
      LabelsType& bootstrapLabels,
      WeightsType& bootstrapWeights);
};

Train a RandomForest with the SequentialBootstrap strategy.

// 1000 random points in 10 dimensions. In reality this might be
// financial time-series data.
arma::mat dataset(10 /* rows */, 1000 /* cols */, arma::fill::randu);

// Random labels for each point, totaling 5 classes.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));

arma::umat intervals(2, labels.n_cols);

for (size_t c = 0; c < 1000; ++c)
{
  // Every "normal" event has length 1 and happens at the time step
  // equivalent to its column.
  intervals(0, c) = c;
  intervals(1, c) = c;
}

// Now set the three "overlapping" events to have longer ranges.
// The first two events overlap in [100,200].
// The third event is isolated from the other two.
// All three still overlap in each time step with one
// "normal" event.
intervals(0, 0) = 0; // start of first event
intervals(1, 0) = 200; // end of first event
intervals(0, 100) = 100; // start of second event
intervals(1, 100) = 500; // end of second event
intervals(0, 600) = 600; // start of third event
intervals(1, 600) = 1000; // end of third event

mlpack::SequentialBootstrap bootstrap(intervals);

// Create and train the random forest.
mlpack::RandomForest<
    mlpack::GiniGain,
    mlpack::MultipleRandomDimensionSelect,
    mlpack::BestBinaryNumericSplit,
    mlpack::AllCategoricalSplit,
    true,
    mlpack::SequentialBootstrap<>> rf(
        dataset,
        labels,
        5, // numClasses
        20, // numTrees
        1, // minimumLeafSize
        1e-7, // minimumGainSplit
        0, // maximumDepth
        mlpack::MultipleRandomDimensionSelect(), // dimSelector
        bootstrap);

std::cout << "Forest trained with sequential bootstrap has " << rf.NumTrees()
    << " trees." << std::endl;