The class is used to split a string into characters. More...

Public Types

using TokenType = int
 The type of the token which the tokenizer extracts. More...

 

Public Member Functions

int operator() (boost::string_view &str) const
 The function extracts the first character from the given string view and removes it from the view. More...

 

Static Public Member Functions

static bool IsTokenEmpty (const int token)
 The function returns true if the given token is equal to EOF. More...

 

Detailed Description

The class is used to split a string into characters.

Definition at line 25 of file char_extract.hpp.

Member Typedef Documentation

◆ TokenType

using TokenType = int

The type of the token which the tokenizer extracts.

Definition at line 29 of file char_extract.hpp.

Member Function Documentation

◆ IsTokenEmpty()

static bool IsTokenEmpty ( const int  token)
inlinestatic

The function returns true if the given token is equal to EOF.

Parameters
tokenThe given token.

Definition at line 56 of file char_extract.hpp.

◆ operator()()

int operator() ( boost::string_view &  str) const
inline

The function extracts the first character from the given string view and removes it from the view.

Each charecter is casted to unsigned char i.e. it belongs to [0, 255]. The functon returns EOF provided that the input string is empty.

Parameters
strString view to retrieve the next token from.

Definition at line 39 of file char_extract.hpp.


The documentation for this class was generated from the following file:
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-3.3.2/src/mlpack/core/data/tokenizers/char_extract.hpp