## π`SparseCoding`

The `SparseCoding` class implements sparse coding with dictionary learning using L1 regularization or elastic net (L1+L2) regularization. Sparse coding is a form of representation learning, and can be used to represent each point in a dataset as a sparse combination of atoms in the dictionary.

#### Simple usage example:

``````// Create a random dataset with 100 points in 40 dimensions, and then a random
// test dataset with 50 points.
arma::mat data(40, 100, arma::fill::randu);
arma::mat testData(40, 50, arma::fill::randu);

// Perform sparse coding with 20 atoms and an L1 penalty of 0.5.
mlpack::SparseCoding sc(20, 0.5);  // Step 1: create object.
double objective = sc.Train(data); // Step 2: learn dictionary.
arma::mat codes;
sc.Encode(testData, codes);        // Step 3: encode new data.

// Print some information about the test encoding.
std::cout << "Average density of encoded test data: "
<< 100.0 * arma::mean(arma::sum(codes != 0)) / codes.n_rows << "\%."
<< std::endl;
``````

More examples...

### π Constructors

• `sc = SparseCoding()`
• `sc = SparseCoding(atoms=0, lambda1=0.0, lambda2=0.0, maxIter=0, objTol=0.01, newtonTol=1e-6)`
• Create a `SparseCoding` object without learning a dictionary on data.
• If `atoms` is set to `0` (the default), it will need to be set to a value greater than `0` before `Train()` is called (`sc.Atoms() = atoms` can be used for this).
• `sc = SparseCoding(data, atoms, lambda1=0.0, lambda2=0.0, maxIter=0, objTol=0.01, newtonTol=1e-6)`
• Create a `SparseCoding` object and train the dictionary on the given `data`.
• The dictionary will contain `atoms` elements.
• `sc = SparseCoding(data, atoms, lambda1, lambda2, maxIter, objTol, newtonTol, initializer)`
• Advanced constructor: create a `SparseCoding` object that will use a custom dictionary initializer and train on the given `data`.
• The dictionary will contain `atoms` elements.
• `initializer` will be used to initialize the dictionary; see Advanced Functionality: Different Dictionary Initialization Strategies for details.

#### Constructor Parameters:

name type description default
`data` `arma::mat` Column-major training matrix. (N/A)
`atoms` `size_t` Number of atoms in dictionary. (N/A)
`lambda1` `double` L1 regularization penalty. Used in both `Train()` and `Encode()` steps. `0.0`
`lambda2` `double` L2 regularization penalty (for elastic net regularization). Used in both `Train()` and `Encode()` steps. `0.0`
`maxIter` `size_t` Maximum number of iterations for dictionary learning. `0` means no limit. `0`
`objTol` `double` Objective function tolerance for terminating dictionary learning. `0.01`
`newtonTol` `double` Tolerance for the Newtonβs method dictionary optimization step. `1e-6`

As an alternative to passing `atoms`, `lambda1`, `lambda2`, `maxIter`, `objTol`, or `newtonTol`, these can be set with a standalone method. The following functions can be used before calling `Train()`:

• `sc.Atoms() = a;` will set the number of atoms to use in the dictionary to `a`. Changing this after calling `Train()` will not make a difference to the dictionary size.

• `sc.Lambda1() = l1;` will set the L1 regularization penalty to `l1`. This can be set after `Train()` to force sparser encodings when `Encode()` is called.

• `sc.Lambda2() = l2;` will set the L2 regularization penalty to `l2`. This can be set after `Train()` to increase the regularization when `Encode()` is called.

• `sc.MaxIterations() = m;` will set the maximum number of iterations for dictionary learning to `m`. `0` means that the algorithm will run until convergence.

• `sc.ObjTolerance() = ot;` will set the objective tolerance for convergence of the dictionary learning algorithm to `ot`.

• `sc.NewtonTolerance() = nt;` will set the tolerance for the Newtonβs method dictionary optimization step to `nt`.

Caveats:

• Larger settings of `atoms` (i.e. larger dictionary sizes) will be able to more accurately represent the data, but may take longer to learn.

• Larger values of `lambda1` will cause the model to use sparser codings for data when `Train()` and `Encode()` are called, but codings that are too sparse may be inaccurate representations of the original points.

• Training is not incremental; a second call to `Train()` will reinitialize the dictionary and restart the learning process.

### π Training

If training the dictionary is not done as part of the constructor call, it can be done with one of the following versions of the `Train()` member function:

• `sc.Train(data)`
• `sc.Train(data, initializer)`
• Train the sparse coding dictionary on the given `data`.
• Optionally, use the given `initializer` to initialize the dictionary (see `DictionaryInitializer` for more details).

### π Encoding

Once a `SparseCoding` model has a trained dictionary, the `Encode()` member function can be used to encode new data points.

• `sc.Encode(data, codes)`
• Encode `data` (a column-major data matrix) as a set of sparse codes of the dictionary, storing the result in `codes`.
• Both `data` and `codes` should be the same matrix type (e.g. `arma::mat`); see Different Element Types for more details.
• `codes` will be set to have `atoms` rows and `data.n_cols` columns.
• Column `i` of `codes` corresponds to the sparse coding of the `i`βth column of `data`. Each row represents the weight associated with each atom in the dictionary.

After encoding, the original data can be recovered (approximately) as `sc.Dictionary() * data`.

### π Other Functionality

• A `SparseCoding` model can be serialized with `data::Save()` and `data::Load()`.

• `sc.Dictionary()` will return an `arma::mat&` containing the dictionary matrix. The matrix has `data.n_rows` rows and `atoms` columns; each column corresponds to an atom in the dictionary.

• `double obj = sc.Objective(data, codes)` computes the sparse coding objective function on the given `data` and encodings `codes`. This can be used after `Encode()` to test the quality of the encodings (a smaller objective is better).

### π Simple Examples

See also the simple usage example for a trivial usage of the `SparseCoding` class.

Train a sparse coding model on the cloud dataset and print the reconstruction error.

``````// See https://datasets.mlpack.org/cloud.csv.
arma::mat dataset;

mlpack::SparseCoding sc;
sc.Atoms() = 50;
sc.Lambda1() = 0.1;
sc.Lambda2() = 0.001;
sc.MaxIterations() = 25;
sc.Train(dataset);

// Encode the training dataset.
arma::mat codes;
sc.Encode(dataset, codes);

std::cout << "Input matrix size: " << dataset.n_rows << " x " << dataset.n_cols
<< "." << std::endl;
std::cout << "Codes matrix size: " << codes.n_rows << " x " << codes.n_cols
<< "." << std::endl;

// Reconstruct the original matrix.
arma::mat recon = sc.Dictionary() * codes;
double error = std::sqrt(arma::norm(dataset - recon, "fro") / dataset.n_elem);
std::cout << "RMSE of reconstructed matrix: " << error << "." << std::endl;
``````

Train a sparse coding model on the iris dataset and save the model to disk.

``````// See https://datasets.mlpack.org/iris.train.csv.
arma::mat dataset;

// Train the model in the constructor.
mlpack::SparseCoding sc(dataset, 10 /* atoms */, 0.1 /* L1 penalty */);

// Save the model to disk.
mlpack::data::Save("sc.bin", "sc", sc);
``````

Load a sparse coding model from disk and encode some new points from the iris dataset.

``````// Load model from disk.
mlpack::SparseCoding sc;

// See https://datasets.mlpack.org/iris.test.csv.
arma::mat dataset;

// Encode the test points.
arma::mat codes;
sc.Encode(dataset, codes);

// Compute the sparse coding objective on the test points.
const double obj = sc.Objective(dataset, codes);
std::cout << "Sparse coding objective on test set: " << obj << "." << std::endl;
``````

Train a sparse coding model on the satellite dataset, trying several different dictionary sizes and checking the objective value on a held-out test dataset.

``````// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat trainData;
// See https://datasets.mlpack.org/satellite.test.csv.
arma::mat testData;

for (size_t atoms = 20; atoms < 100; atoms += 10)
{
mlpack::SparseCoding sc(atoms);
sc.Lambda1() = 0.1;
sc.MaxIterations() = 20; // Keep iterations low so this runs relatively fast.

const double trainObj = sc.Train(trainData);

// Compute the objective on the test set.
arma::mat codes;
sc.Encode(testData, codes);
const double testObj = sc.Objective(testData, codes);

std::cout << "Atoms: " << std::setfill(' ') << std::setw(3) << atoms << "; ";
std::cout << "training set objective: " << std::setw(6) << trainObj << "; ";
std::cout << "test set objective: " << std::setw(6) << testObj << "."
<< std::endl;
}
``````

### π Advanced Functionality: Template Parameters

The `SparseCoding` class has one class template parameter that can be used for custom behavior. The full signature of the class is:

``````SparseCoding<MatType>
``````

In addition, the constructors and `Train()` functions have a template parameter `DictionaryInitializer` that can be used for custom behavior.

• `MatType`: the type of the matrix to use (e.g. `arma::mat`, `arma::fmat`, etc.). The given `MatType` must support the Armadillo API and hold a floating-point element type (e.g. `float`, `double`, etc.).

• `DictionaryInitializer`: the strategy used to initialize the dictionary. By default, `DataDependentRandomInitializer` is used.

#### `MatType`: Different Element Types

`MatType` specifies the type of matrix used for training data and internal representation of the dictionary. Any matrix type that implements the Armadillo API can be used. The example below trains a sparse coding model on 32-bit floating point data.

``````// See https://datasets.mlpack.org/cloud.csv.
arma::fmat dataset;

mlpack::SparseCoding<arma::fmat> sc;
sc.Atoms() = 30;
sc.Lambda1() = 0.15;
sc.Lambda2() = 0.001;
sc.MaxIterations() = 100;
// Note: a looser tolerance is often required when using floats instead of
// doubles.
sc.ObjTolerance() = 0.01;
sc.Train(dataset);

// Encode the training dataset.
arma::fmat codes;
sc.Encode(dataset, codes);

std::cout << "Input matrix size: " << dataset.n_rows << " x " << dataset.n_cols
<< "." << std::endl;
std::cout << "Codes matrix size: " << codes.n_rows << " x " << codes.n_cols
<< "." << std::endl;

// Reconstruct the original matrix.
arma::fmat recon = sc.Dictionary() * codes;
double error = std::sqrt(arma::norm(dataset - recon, "fro") / dataset.n_elem);
std::cout << "RMSE of reconstructed matrix: " << error << "." << std::endl;
``````

#### `DictionaryInitializer`: Different Dictionary Initialization Strategies

The `DictionaryInitializer` template class specifies the strategy to be used to initialize the dictionary when `Train()` is called.

• The `DataDependentRandomInitalizer` class (the default) uses the average of three random points in the dataset to initialize each atom in the dictionary.

• The `NothingInitializer` class does not modify the dictionary matrix in any way, and could be used either to set a specific dictionary before training with `sc.Dictionary()`, or to allow incremental training that does not modify the existing dictionary when `Train()` is called a second time.

• The `RandomInitializer` class initializes the dictionary by sampling norm-1 atoms from a normal distribution.

Note: none of the classes above have any members, and as such it is not necessary to use the constructor or `Train()` variants that take an initialized `initializer` object. That would only be necessary for a custom `DictionaryInitializer` class that stored internal members.

The example below uses `NothingInitializer` to set a specific initial dictionary.

``````// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat trainData;

const size_t atoms = 25;
const double lambda1 = 0.1;
const double lambda2 = 0.0;
const size_t maxIterations = 50;

// Use a uniform random matrix as the initial dictionary.
arma::mat initialDictionary(trainData.n_rows, atoms, arma::fill::randu);

mlpack::SparseCoding sc(atoms, lambda1, lambda2, maxIterations);
sc.Dictionary() = initialDictionary;

const double obj = sc.Train<mlpack::NothingInitializer>(trainData);
std::cout << "Training set objective: " << obj << "." << std::endl;
``````

• An entirely custom class can also be implemented. The class must implement one method, `Initialize()`:
``````// You can use this as a starting point for implementation.
class CustomDictionaryInitializer
{
public:
// Initialize the dictionary to have the given number of atoms, given the
// dataset.  MatType will be the matrix type used by the sparse coding model
// (e.g. `arma::mat`, `arma::fmat`, etc.).
template<typename MatType>
void Initialize(const MatType& data,
const size_t atoms,
MatType& dictionary);
};
``````