Boost C++ Libraries

...one of the most highly regarded and expertly designed C++ library projects in the world. — Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

This is the documentation for an old version of boost. Click here for the latest Boost documentation.

C++ Boost

Tokenizer Class

  template <
        class TokenizerFunc = char_delimiters_separator<char>, 
        class Iterator = std::string::const_iterator,
        class Type = std::string
  >
  class tokenizer

The tokenizer class provides a container view of a series of tokens contained in a sequence. You set the sequence to parse and the TokenizerFunction to use to parse the sequence either upon construction or using the assign member function. Note: No parsing is actually done upon construction. Parsing is done on demand as the tokens are accessed via the iterator provided by begin.

Example

// simple_example_1.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>

int main(){
   using namespace std;
   using namespace boost;
   string s = "This is,  a test";
   tokenizer<> tok(s);
   for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
       cout << *beg << "\n";
   }
}

The output from simple_example_1 is:

This is a test

Template Parameters

Parameter	Description
`TokenizerFunc`	The TokenizerFunction used to parse the sequence.
`Iterator`	The type of the iterator the specifies the sequence.
`Type`	The type of the token, typically string.

Related Types

Type	Remarks
iterator	The type returned by begin and end. Note: the category of iterator will be at most ForwardIterator. It will be InputIterator if the Iterator template parameter is an InputIterator. For any other category, it will be ForwardIterator.
const_iterator	Same type as iterator.
value_type	Same type as the template parameter Type
reference	Same type as value_type&
const_reference	Same type as const reference
pointer	Same type as value_type*
const_pointer	Same type as const pointer
size_type	void
difference_type	void

Construction and Member Functions

tokenizer(Iterator first, Iterator last,const TokenizerFunc& f = TokenizerFunc()) 

template<class Container>
tokenizer(const Container& c,const TokenizerFunc& f = TokenizerFunc())

void assign(Iterator first, Iterator last)

void assign(Iterator first, Iterator last, const TokenizerFunc& f)

template<class Container>
void assign(const Container& c)

template<class Container>
void assign(const Container& c, const TokenizerFunc& f)

iterator begin() const 

iterator end() const

Parameter	Description
c	A container that contains the sequence to parse. Note: c.begin() and c.end() must be convertible to the template parameter Iterator.
f	A functor that is a model of TokenizerFunction that will be used to parse the sequence.
first	The iterator that represents the beginning position in the sequence to be parsed.
last	The iterator that represents the past the end position in the sequence to be parsed.

Revised 16 February, 2008

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)