Getting started with Boost.Metaparse

This tutorial shows you how to build a parser for a small calculator language from the ground up. The goal is not to have a complete calculator, but to show you the most common situations one can face while building a parser using Metaparse. This tutorial assumes, that you have some template metaprogramming experience.

1.1. Testing environment

While you are using Metaparse, you will be writing parsers turning an input text into a type. These types can later be processed by further template metaprograms. While you are working on your parsers, you'll probably want to look at the result of parsing a test input. This tutorial assumes that you can use Metashell. Since the online demo makes the Boost headers available, you can use that in the tutorial as well.

If you install Metashell on your computer, make sure that you have the Boost libraries and the getting_started example of Metaparse on the include path. For example, you can start Metashell with the following arguments:

$ metashell -I$BOOST_ROOT -I$BOOST_ROOT/libs/metaparse/example/getting_started

$BOOST_ROOT refers to the boost root directory (where you have checked out the Boost source code).

This tutorial is long and therefore you might want to make shorter or longer breaks while reading it. To make it easy for you to stop at a certain point and continue later (or to start in the middle if you are already familiar with the basics) Metaparse has a getting_started directory in the examples. This contains the definitions for each section of this tutorial.

If you're about to start (or continue) this guide from section 5.2.1, you can include 5_2_1.hpp. This will define everything you need to start with that section.

	Note
	You have access to these headers in the online Metashell demo as well. For example you can include the `<boost/metaparse/getting_started/5_2_1.hpp>` header to start from section 5.2.1.

1.2. Using a "regular" testing environment

1.2.1. Enforcing an error message or a warning containing the result of the metafunction call
1.2.2. Displaying the result of the metafunction call at runtime

If you have no access to Metashell or you prefer using your regular C++ development environment while processing this tutorial, this is also possible.

The tutorial (and usually experimenting with Metaparse) requires that you evaluate different template metaprogramming expressions and check their result, which is a type. Thus, to try the examples of this tutorial you need a way to be able to display the result of evaluating a template metaprogram. This section shows you two options.

1.2.1. Enforcing an error message or a warning containing the result of the metafunction call

You can either use boost::mpl::print or mpllibs::metamonad::fail_with_type to enforce a warning or an error message containing the result of a metaprogram evaluation. For example to see what BOOST_METAPARSE_STRING("11 + 2") refers to, you can create a test.cpp with the following content:

#include <boost/metaparse/string.hpp>
#include <boost/mpl/print.hpp>

boost::mpl::print<BOOST_METAPARSE_STRING("11 + 2")> x;

If you try to compile it, the compiler will display warnings containing the type the expression BOOST_METAPARSE_STRING("11 + 2") constructs. To use this technique for this tutorial, you need to add all the includes and definitions the tutorial suggests typing in the shell to your test.cpp file. When the shell suggests to try to call some metafunction (or you'd like to try something out), you need to replace the template argument of boost::mpl::print with the expression in question and recompile the code.

1.2.2. Displaying the result of the metafunction call at runtime

You can also display the result of metaprograms at runtime. You can use the Boost.TypeIndex library to do this. For example to see what BOOST_METAPARSE_STRING("11 + 2") refers to, you can create a test.cpp with the following content:

#include <boost/metaparse/string.hpp>
#include <boost/type_index.hpp>
#include <iostream>

int main()
{
  std::cout
    << boost::typeindex::type_id_with_cvr<BOOST_METAPARSE_STRING("11 + 2")>()
    << std::endl;
}

If you compile and run this code, it will display the type on the standard output.

2. The text to parse

With Metaparse you can create template metaprograms parsing an input text. To pass the input text to the metaprograms, you need to represent them as types. For example let's represent the text "Hello world" as a type. The most straightforward way of doing it would be creating a variadic template class taking the characters of the text as template arguments:

template <char... Cs>
struct string;

The text "11 + 2" can be represented the following way:

string<'1', '1', ' ', '+', ' ', '2'>

Metaparse provides this type for you. Run the following command in Metashell:

> #include <boost/metaparse/string.hpp>

	Note
	Note that the `>` character at the beginning of the above code example is the prompt of Metashell. It is added to the code examples as a hint to what you should run in Metashell (or add to your test `cpp` file if you are using a regular development environment).

	Note
	Note that in the online-demo of Metashell you can paste code into the shell by right-clicking on the shell somewhere and choosing Paste from browser in the context menu.

This will make this type available for you. Now you can try running the following command:

> boost::metaparse::string<'1', '1', ' ', '+', ' ', '2'>

The shell will echo (almost) the same type back to you. The only difference is that it is in a sub-namespace indicating the version of Metaparse being used.

The nice thing about this representation is that metaprograms can easily access the individual characters of the text. The not so nice thing about this representation is that if you want to write the text "Hello world" in your source code, you have to type a lot.

Metaparse provides a macro that can turn a string literal into an instance of boost::metaparse::string. This is the BOOST_METAPARSE_STRING macro. You get it by including <boost/metaparse/string.hpp>. Let's try it by running the following command in Metashell:

> BOOST_METAPARSE_STRING("11 + 2")

You will get the same result as you got by instantiating boost::metaparse::string yourself.

3. Creating a simple parser

3.1. Dealing with invalid input
3.2. Dealing with input containing more than what is needed
3.3. Accepting optional whitespaces at the end of the input

	Note
	Note that you can find everything that has been included and defined so far here.

Let's try creating a parser. We will start with creating a parser for something simple: we will be parsing integer numbers, such as the text "13". You can think of this first parsing exercise as a template metaprogramming string-to-int conversion because we expect to get the value 13 as the result of parsing.

	Note
	You know the difference between `"13"` and `13` in C++. One of them is a character array, the other one is an integral value. But what is the difference between them in template metaprogramming? They are represented by different types. For example `"13"` is represented by `string<'1', '3'>` while `13` is represented by `std::integral_constant<int, 13>`.

To build a parser, we need to specify the grammar to use. Metaparse provides building blocks (called parsers) we can use to do this and one of them is the int_ parser which does exactly what we need: it parses integers. To make it available, we need to include it:

> #include <boost/metaparse/int_.hpp>

Our grammar is simple: int_. (Don't worry, we'll parse more complicated languages later).

A parser is a template metafunction class. It can be used directly, but its interface is designed for completeness and not for ease of use. Metaparse provides the build_parser metafunction that adds a wrapper to parsers with a simple interface.

	Note
	In this tutorial, we will always be wrapping our parsers with this. We will call these wrapped parsers parsers as well. If you are interested in it, you can learn about the complete interface of parsers here.

Let's create a parser using int_ and build_parser:

> #include <boost/metaparse/build_parser.hpp>
> using namespace boost::metaparse;
> using exp_parser1 = build_parser<int_>;

	Note
	Note that if the last character of your command is the `\` character in Metashell, then the shell assumes that you will continue typing the same command and waits for that before evaluating your command. When Metashell is waiting for the second (or third, or fourth, etc) line of a command, it uses a special prompt, `...>`.

	Note
	Note that we have to use `boost::mpl::quote1` to turn our `eval_plus` metafunction into a metafunction class.

	Note
	Note that this is very similar to the `std::accumulate` algorithm. Boost.MPL provides `boost::mpl::accumulate` as well, which is a synonym for `boost::mpl::fold`. This tutorial (and Metaparse) uses the name `fold`.

	Note
	We are using `sum_items` as the function to call in each iteration. We are passing a metafunction (`sum_items`) to another metafunction (`boost::mpl::fold`) as an argument. To be able to do this, we need to turn it into a template metafunction class using `boost::mpl::quote2` (`2` means that it takes two arguments).

	Note
	Note that we have been talking about loops while there is no such thing as a loop in template metaprogramming. Loops can be implemented using recursion: every recursive call is one iteration of the loop. The loop is stopped at the bottom of the recursive chain.

	Note
	You can give any parser to `one_of`, therefore it is possible that more than one of them can parse the input. In those cases the order matters: `one_of` tries parsing the input with the parsers from left to right and the first one that succeeds, wins.

	Note
	Note that it does not satisfy the expectations of a template metafunction since it takes the operator as a `char` and not as a `class` (or `typename`) argument. For simplicity, we will still call it a metafunction.

	Note
	Note that folding solutions going from left to right is implemented in a more efficient way than folding from right to left. Therefore when both solutions can be used you should prefer folding from left to right.

	Note
	Since we are versioning the different parser implementations in Metashell (`paren_exp1`, `paren_exp2`, etc) you might try to define these recursive parsers and it might seem to work for the first time. In that case, when you later try creating a parser as part of a library (save your Metashell environment to a file or re-implement the important/successful elements) you face this issue.