mlpack

Image preprocessing

mlpack provides a set of functions to preprocess images for training and testing.

🔗 Resizing images

The ResizeImages() function can be used to resize image data:

Example usage of the ResizeImages() function on a set of images with different dimensions:

// See https://datasets.mlpack.org/sheep.tar.bz2
arma::Mat<unsigned char> image;
mlpack::ImageOptions opts;
opts.Fatal() = false;

// The images are located in our test/data directory. However, any image could
// be used instead.
std::vector<std::string> files =
    {"sheep_1.jpg", "sheep_2.jpg", "sheep_3.jpg", "sheep_4.jpg",
     "sheep_5.jpg", "sheep_6.jpg", "sheep_7.jpg", "sheep_8.jpg",
     "sheep_9.jpg"};

// The resized images will be saved locally. We are declaring the vector that
// contains the names of the resized images.
std::vector<std::string> reSheeps =
    {"re_sheep_1.jpg", "re_sheep_2.jpg", "re_sheep_3.jpg", "re_sheep_4.jpg",
     "re_sheep_5.jpg", "re_sheep_6.jpg", "re_sheep_7.jpg", "re_sheep_8.jpg",
     "re_sheep_9.jpg"};

// Load and Resize each one of them individually, because they do not have
// the same dimensions. The `opts` will contain the dimension for each one.
for (size_t i = 0; i < files.size(); i++)
{
  mlpack::Load(files.at(i), image, opts);
  mlpack::ResizeImages(image, opts, 320, 320);
  mlpack::Save(reSheeps.at(i), image, opts);
}

Example usage of ResizeImages() function on a set of images that have the same dimensions.

// All images have the same dimension, It would be possible to load all of
// them into one matrix

// See https://datasets.mlpack.org/sheep.tar.bz2
arma::Mat<unsigned char> images;
mlpack::ImageOptions opts;
opts.Fatal() = false;

std::vector<std::string> reSheeps =
    {"re_sheep_1.jpg", "re_sheep_2.jpg", "re_sheep_3.jpg", "re_sheep_4.jpg",
     "re_sheep_5.jpg", "re_sheep_6.jpg", "re_sheep_7.jpg", "re_sheep_8.jpg",
     "re_sheep_9.jpg"};

mlpack::Load(reSheeps, images, opts);

// Now let us resize all these images at once, to specific dimensions.
mlpack::ResizeImages(images, opts, 160, 160);

// The resized images will be saved locally. We are declaring the vector that
// contains the names of the resized images.
std::vector<std::string> smSheeps =
    {"sm_sheep_1.jpg", "sm_sheep_2.jpg", "sm_sheep_3.jpg", "sm_sheep_4.jpg",
     "sm_sheep_5.jpg", "sm_sheep_6.jpg", "sm_sheep_7.jpg", "sm_sheep_8.jpg",
     "sm_sheep_9.jpg"};

mlpack::Save(smSheeps, images, opts);

🔗 Resize and crop images

In addition to resizing images, mlpack also provides resize-and-crop functionality. This is useful when the desired aspect ratio of an image differs largely from the original image.

The resize-and-crop operation, given a target size outputWidth x outputHeight, first resizes the image while preserving the aspect ratio such that the width and height of the image both no smaller than outputWidth and outputHeight. Then, the image is cropped to have size outputWidth by outputHeight, keeping the center pixels only. This process is shown below.

Original image:

cat

Original image with target size of 220x220 pixels:

cat with rectangle overlaid

First step: resize while preserving aspect ratio:

scaled cat with rectangle overlaid

Second step: crop to desired final size:

cropped cat

Example usage of the ResizeCropImages() function on a set of images with different dimensions:

// See https://datasets.mlpack.org/sheep.tar.bz2.
arma::Mat<unsigned char> image;
mlpack::ImageOptions opts;
opts.Fatal() = false;

// The images are located in our test/data directory. However, any image could
// be used instead.
std::vector<std::string> files =
    {"sheep_1.jpg", "sheep_2.jpg", "sheep_3.jpg", "sheep_4.jpg",
     "sheep_5.jpg", "sheep_6.jpg", "sheep_7.jpg", "sheep_8.jpg",
     "sheep_9.jpg"};

// The resized images will be saved locally. We are declaring the vector that
// contains the names of the resized and cropped images.
std::vector<std::string> cropSheeps =
    {"crop_sheep_1.jpg", "crop_sheep_2.jpg", "crop_sheep_3.jpg",
     "crop_sheep_4.jpg", "crop_sheep_5.jpg", "crop_sheep_6.jpg",
     "crop_sheep_7.jpg", "crop_sheep_8.jpg", "crop_sheep_9.jpg"};

// Load and resize-and-crop each image individually, because they do not have
// the same dimensions. The `opts` will contain the dimension for each one.
for (size_t i = 0; i < files.size(); i++)
{
  mlpack::Load(files.at(i), image, opts);
  mlpack::ResizeCropImages(image, opts, 320, 320);
  mlpack::Save(cropSheeps.at(i), image, opts);
  std::cout << "Resized and cropped " << files.at(i) << " to "
      << cropSheeps.at(i) << " with output size 320x320." << std::endl;
}

🔗 Changing the memory layout of images

When loading images using Load() channels are interleaved, i.e. the underlying vector contains the values [r, g, b, r, g, b, ... ] (for an image with 3 channels). mlpack has functionality such as Convolution that requires channels be grouped, e.g [r, r, ..., g, g, ..., b, b]. The same is true when using Save(), the channels are expected to be interleaved.

To convert the layout of your image from interleaved channels to grouped channels and vice versa, you can use GroupChannels() and InterleaveChannels().

NOTE: Other image related functions (such as ResizeImages etc) require channels be interleaved. If you need to use GroupChannels() make sure to resize or crop your images first beforehand.

GroupChannels()

InterleaveChannels()

Example

This example loads an image, and converts the layout such that channels are grouped together in preparation for a convolutional neural network. Then, the image is converted back to interleaved channels and saved.

// Download: https://datasets.mlpack.org/images/mlpack-favicon.png
arma::mat image;
mlpack::ImageOptions opts;
opts.Fatal() = true;
mlpack::Load("mlpack-favicon.png", image, opts);

std::vector<std::string> colors =
     { "\033[31m", "\033[32m", "\033[34m", "\033[37m" };

// Display input before grouping channels (Load() returns channels interleaved).
std::cout << "Original Image (channels interleaved):" << std::endl;
for (size_t i = 0; i < image.n_rows; i += opts.Channels())
{
  for (size_t j = 0; j < opts.Channels(); j++)
    std::cout << colors[j] << image.at(i + j, 0) << "\033[0m" << ", ";
}
std::cout << std::endl << std::endl;

// Group channels.
image = mlpack::GroupChannels(image, opts);

// Display submatrix of input after grouping channels
std::cout << "Grouped channels:" << std::endl;
for (size_t i = 0; i < opts.Channels(); i++)
{
  for (size_t j = 0; j < image.n_rows / opts.Channels(); j++)
    std::cout << colors[i] <<
      image.at(i * image.n_rows / opts.Channels() + j, 0) << "\033[0m" << ", ";
}
std::cout << std::endl << std::endl;

// Do some computation here; for example, a convolutional neural network.

// Interleave channels to prepare for saving.
image = mlpack::InterleaveChannels(image, opts);

// Display input after interleaving channels.
// This should be identical to the original image.
std::cout << "Interleaved channels (identical to original):" << std::endl;
for (size_t i = 0; i < image.n_rows; i += opts.Channels())
{
  for (size_t j = 0; j < opts.Channels(); j++)
    std::cout << colors[j] << image.at(i + j, 0) << "\033[0m" << ", ";
}
std::cout << std::endl << std::endl;

mlpack::Save("mlpack-favicon.png", image, opts);

🔗 Letterbox transform

The letterbox transform resizes an image’s dimensions to width x height but keeps the aspect ratio of the original image. Whitespace is then filled in with fillValue.

Original image with size of 640x326 pixels:

cat

Image with target size of 416x416 pixels after letterbox:

cat with square letterbox transform

Image with target size of 300x208 pixels after letterbox:

cat with rectangular letterbox transform

Example

An example that loads an image, resizes the image to some square image while keeping the aspect ratio using LetterboxImages().

// Download: https://datasets.mlpack.org/jurassic-park.png
arma::mat image;
mlpack::ImageOptions opts;
opts.Fatal() = true;
mlpack::Load("jurassic-park.png", image, opts);
mlpack::LetterboxImages(image, opts, 416, 416, 127.0);
// Image dimensions are now 416x416.
mlpack::Save("jurassic-park-letterbox.png", image, opts);

std::cout << "Dimensions: " << opts.Width() << " x " << opts.Height()
          << " x " << opts.Channels() << "\n";
std::cout << "Total size: " << image.n_rows << "\n";

🔗 Draw bounding boxes for object detection

mlpack comes with a utility function to draw bounding boxes onto images when doing tasks such as object detection.

jurassic park logo

You can do this through the BoundingBoxImage() function.

BoundingBoxImage()

Example

An example that draws a red bounding box onto an image, with the class name Jurassic Park Logo.

// See https://datasets.mlpack.org/jurassic-park.png
arma::mat image;
mlpack::data::ImageOptions opts;
opts.Fatal() = true;
mlpack::data::Load("jurassic-park.png", image, opts);

arma::vec bbox({90, 80, 510, 370});
arma::vec color({255, 0, 0}); // Use red for the border.
std::string className = "Jurassic Park Logo";
size_t borderSize = 2;
size_t letterSize = 2;
mlpack::data::BoundingBoxImage(image, opts, bbox, color, borderSize,
                               className, letterSize);
mlpack::data::Save("jurassic-park-box.png", image, opts, true);