The SplitByAnyOf class tokenizes a string using a set of delimiters. More...

Public Types

using MaskType = std::array< bool, 1<< CHAR_BIT >
 A convenient alias for the mask type. More...

 
using TokenType = boost::string_view
 The type of the token which the tokenizer extracts. More...

 

Public Member Functions

 SplitByAnyOf (const boost::string_view delimiters)
 Construct the object from the given delimiters. More...

 
const MaskTypeMask () const
 Return the mask. More...

 
MaskTypeMask ()
 Modify the mask. More...

 
boost::string_view operator() (boost::string_view &str) const
 The function extracts the first token from the given string view and then removes the prefix containing the token from the view. More...

 

Static Public Member Functions

static bool IsTokenEmpty (const boost::string_view token)
 The function returns true if the given token is empty. More...

 

Detailed Description

The SplitByAnyOf class tokenizes a string using a set of delimiters.

Definition at line 26 of file split_by_any_of.hpp.

Member Typedef Documentation

◆ MaskType

using MaskType = std::array<bool, 1 << CHAR_BIT>

A convenient alias for the mask type.

Definition at line 33 of file split_by_any_of.hpp.

◆ TokenType

using TokenType = boost::string_view

The type of the token which the tokenizer extracts.

Definition at line 30 of file split_by_any_of.hpp.

Constructor & Destructor Documentation

◆ SplitByAnyOf()

SplitByAnyOf ( const boost::string_view  delimiters)
inline

Construct the object from the given delimiters.

Parameters
delimitersThe given delimiters.

Definition at line 40 of file split_by_any_of.hpp.

Member Function Documentation

◆ IsTokenEmpty()

static bool IsTokenEmpty ( const boost::string_view  token)
inlinestatic

The function returns true if the given token is empty.

Parameters
tokenThe given token.

Definition at line 78 of file split_by_any_of.hpp.

◆ Mask() [1/2]

const MaskType& Mask ( ) const
inline

Return the mask.

Definition at line 84 of file split_by_any_of.hpp.

◆ Mask() [2/2]

MaskType& Mask ( )
inline

Modify the mask.

Definition at line 86 of file split_by_any_of.hpp.

◆ operator()()

boost::string_view operator() ( boost::string_view &  str) const
inline

The function extracts the first token from the given string view and then removes the prefix containing the token from the view.

Parameters
strString view to retrieve the token from.

Definition at line 54 of file split_by_any_of.hpp.


The documentation for this class was generated from the following file:
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-3.3.2/src/mlpack/core/data/tokenizers/split_by_any_of.hpp