Boost.Jam

Introduction

Features

Contents

Building Boost.Jam

Core Jam Extensions

Command-line and Environment Variable Quoting

Startup Behavior

Rule Indirection

Argument Lists

Module Support

Declaration

Variable Scope

Local Rules

The RULENAMES rule

The VARNAMES rule

The IMPORT rule

The EXPORT rule

The CALLER_MODULE

The DELETE_MODULE rule

Local for Loop Variables

Negative Indexing

Support for Cygwin

Target Binding Detection

Return Code Inversion

Ignoring Return Codes

Removing outdated targets

The SUBST Rule

The JAM_VERSION global variable

Debugging Support

The BACKTRACE rule

Profiling

Parser Debugging

Dependency Graph Output

The UPDATE rule and changes to command line handling

Semaphores

The W32_GETREG rule

Jam Fundamentals

Introduction

Boost.Jam (BJam)  is a build tool based on FTJam, which in turn is based on Perforce Jam. It contains significant improvements made to facilitate its use in the Boost Build System, but should be backward compatible with Perforce Jam.

This is version 3.1.10 of BJam and is based on version 2.4 of Jam/MR:

/+\
+\  Copyright 1993-2002 Christopher Seiwald and Perforce Software, Inc.
\+/
This is Release 2.4 of Jam/MR, a make-like program.
License is hereby granted to use this software and distribute it
freely, as long as this copyright notice is retained and modifications
are clearly marked.
ALL WARRANTIES ARE HEREBY DISCLAIMED.

Features

Jam is a make(1) replacement that makes building simple things simple and building complicated things manageable.

Jam's language is expressive, making Jamfiles (c.f. Makefiles) compact. Here's a sample:

Main smail : main.c map.c resolve.c deliver.c
     misc.c parser.y alias.c pw.c headers.c
     scanner.l getpath.c str.c ;

This builds "smail" from a dozen source files. Jam handles header file dependencies automatically and on-the-fly.

Jam is very portable: it runs on UNIX, VMS, Mac, and NT. Most Jamfiles themselves are portable, like the sample above.

Jam is unintrusive: it is small, it has negligible CPU overhead, and it doesn't create any of its own funny files (c.f. Odin, nmake, SunOS make).

Jam can build large projects spread across many directories in one pass, without recursing, tracking the relationships among all files. Jam can do this with multiple, concurrent processes.

Jam isn't under the blinkin GNU copyright, so you can incorporate it into commercial products.

Contents

Building Boost.Jam

Installing BJam after building it is simply a matter of copying the generated executables someplace in your PATH. For building the executables there are a set of build bootstrap scripts to accomodate particular environments. The scripts take one optional argument, the name of the toolset to build with. When the toolset is not given an attempt is made to detect an available toolset and use that. The build scripts accept these areguments:

<build script name> [toolset]

Running the scripts without arguments will give you the best chance of success. On Windows platforms from a command console do:

cd <jam source location>
.\build.bat

On Unix type platforms do:

cd <jam source location>
sh ./build.sh

For the Boost.Jam source included with the Boost distribution the <jam source location> is BOOST_ROOT/tools/build/jam_src.

If the scripts fail to detect an appropriate toolset to build with your particular toolset may not be auto-detectable. In that case, you can specify the toolset as the first argument, this assumes that the toolset is readily available in the PATH. NOTE: The toolset used to build Boost.Jam is independent of the toolsets used for Boost.Build. Only one version of Boost.Jam is needed to use Boost.Build. The supported toolsets, and wether they are auto-detected, are:

The built executables are placed in a subdirectory specific to your platform. For example, in Linux running on an Intel x86 compatible chip, the executables are placed in: "bin.linuxx86". There are two executables generated: jam[.exe], and bjam[.exe], both are the same binary but with different names. The "jam" invocation is used for compatability with the Perforce Jam/MR functionality, whereas "bjam" is used for the extended Boost.Build functionality.

The build scripts support additional invocation arguments for use by developers of Boost.Jam. The extra arguments come after the toolset, and can take the form of "--option" or targets for the build.jam script:

<build script name> [toolset] [--option+ target*]

There is current only one available option, "--debug", which builds debugging versions of the executable. When built they are placed in their own directory "bin.<platform>.debug". To specify targets without options, one can suply a special ignore option "---".

Currently there are two targets supported: dist, and clean. Respectively they: generate packages (compressed archives) as appropriate for distribution in the platform, or remove all the built executables and objects.

Core Jam Extensions

A number of enhancements have been made to the core language of Classic Jam. These changes were aimed primarily at making it easier to manage the complexity of a large system such as Boost.Build.

Command-line and Environment Variable Quoting

Classic Jam had an odd behavior with respect to command-line variable ( -s...) and environment variable settings which made it impossible to define an arbitrary variable with spaces in the value. Boost Jam remedies that by treating all such settings as a single string if they are surrounded by double-quotes. Uses of this feature can look interesting, since shells require quotes to keep characters separated by whitespace from being treated as separate arguments:

jam -sMSVCNT="\"\"C:\Program Files\Microsoft Visual C++\VC98\"\"" ...

The outer quote is for the shell. The middle quote is for Jam, to tell it to take everything within those quotes literally, and the inner quotes are for the shell again when paths are passed as arguments to build actions. Under NT, it looks a lot more sane to use environment variables before invoking jam when you have to do this sort of quoting:

set MSVCNT=""C:\Program Files\Microsoft Visual C++\VC98\""

Startup Behavior

The Boost.Build v2 initialization behavior has been implemented. This behavior only applies when the executable being invoked is called "bjam" or, for backward-compatibility, when the BOOST_ROOT variable is set.

1. We attempt to load "boost-build.jam" by searching from the current invocation directory up to the root of the file-system. This file is expected to invoke the boost-build rule to indicate where the Boost.Build system files are, and to load them.
2. If boost-build.jam is not found we error and exit, giving brief instructions on possible errors.

   As a backward-compatibility measure for older versions of Boost.Build, when the BOOST_ROOT variable is set, we first search for boost-build.jam in $(BOOST_ROOT)/tools/build and $(BOOST_BUILD_PATH). If found, it is loaded and initialization is complete.

3. The boost-build rule adds its (optional) argument to the front of BOOST_BUILD_PATH, and attempts to load bootstrap.jam from those directories. If a relative path is specified as an argument, it is treated as though it was relative to the boost-build.jam file.
4. If the bootstrap.jam file was not found, we print a likely error message and exit.

Rule Indirection

Boost Jam allows you to call a rule whose name is held in a variable or computed as the result of an expression:

x = foo ;
rule foobar { ECHO foobar ; }   # a trivial rule
$(x)bar ;                       # invokes foobar

Furthermore, if the first expression expands to more than one list item, everything after the first item becomes part of the first argument. This allows a crude form of argument binding:

# return the elements of sequence for which predicate returns non-nil
rule filter ( sequence * : predicate + )
{
    local result ;
    for local x in $(sequence)
    {
        if [ $(predicate) $(x) ] { result += $(x); }
    }
    return $(result);
}
# true iff x == y
rule equal ( x y )
{
    if $(x) = $(y) { return true; }
}
# bind 3 to the first argument of equal
ECHO [ filter 1 2 3 4 5 4 3 : equal 3 ] ; # prints "3 3"

Argument lists

You can now describe the arguments accepted by a rule, and refer to them by name within the rule. For example, the following prints ``I'm sorry, Dave'' to the console:

rule report ( pronoun index ? : state : names + )
{
    local he.suffix she.suffix it.suffix = s ;
    local I.suffix = m ;
    local they.suffix you.suffix = re ;
    ECHO $(pronoun)'$($(pronoun).suffix) $(state), $(names[$(index)]) ;
}
report I 2 : sorry : Joe Dave Pete ;

Each name in a list of formal arguments (separated by ``:'' in the rule declaration) is bound to a single element of the corresponding actual argument unless followed by one of these modifiers:

The actual and formal arguments are checked for inconsistencies, which cause Jam to exit with an error code:

### argument error
# rule report ( pronoun index ?  : state  : names + )
# called with: ( I 2 foo  : sorry  : Joe Dave Pete )
# extra argument foo
### argument error
# rule report ( pronoun index ?  : state  : names + )
# called with: ( I 2  : sorry )
# missing argument names

If you omit the list of formal arguments, all checking is bypassed as in ``classic'' Jam. Argument lists drastically improve the reliability and readability of your rules, however, and are strongly recommended for any new Jam code you write.

Module Support

Boost Jam introduces support for modules, which provide some rudimentary namespace protection for rules and variables. A new keyword, ``module'' was also introduced. The features described in this section are primitives, meaning that they are meant to provide the operations needed to write Jam rules which provide a more elegant module interface.

Declaration

module expression { ... }

Code within the { ... } executes within the module named by evaluating expression. Rule definitions can be found in the module's own namespace, and in the namespace of the global module as module-name.rule-name, so within a module, other rules in that module may always be invoked without qualification:

module my_module
{
    rule salute ( x ) { ECHO $(x), world ; }
    rule greet ( ) { salute hello ; }
    greet ;
}
my_module.salute goodbye ;

When an invoked rule is not found in the current module's namespace, it is looked up in the namespace of the global module, so qualified calls work across modules:

module your_module
{
    rule bedtime ( ) { my_module.salute goodnight ; }
}

Variable Scope

Each module has its own set of dynamically nested variable scopes. When execution passes from module A to module B, all the variable bindings from A become unavailable, and are replaced by the bindings that belong to B. This applies equally to local and global variables:

module A
{
    x = 1 ;
    rule f ( )
    {
        local y = 999 ; # becomes visible again when B.f calls A.g
        B.f ;
    }
    rule g ( )
    {
        ECHO $(y) ;     # prints "999"
    }
}
module B
{
    y = 2 ;
    rule f ( )
    {
        ECHO $(y) ; # always prints "2"
        A.g ;
    }
}

The only way to access another module's variables is by entering that module:

rule peek ( module-name ? : variables + )
{
    module $(module-name)
    {
        return $($(>)) ;
    }
}

Local Rules

local rule rulename...

The rule is declared locally to the current module. It is not entered in the global module with qualification, and its name will not appear in the result of:

[ RULENAMES module-name ]

The RULENAMES Rule

rule RULENAMES ( module ? )

Returns a list of the names of all non-local rules in the given module. If module is omitted, the names of all non-local rules in the global module are returned.

The VARNAMES Rule

rule VARNAMES ( module ? )

Returns a list of the names of all variable bindings in the given module. If module is omitted, the names of all variable bindings in the global module are returned. Note:this includes any local variables in rules from the call stack which have not returned at the time of the VARNAMES invocation.

The IMPORT Rule

IMPORT allows rule name aliasing across modules:

rule IMPORT ( source_module ? : source_rules *
            : target_module ? : target_rules * )

The IMPORT rule copies rules from the source_module into the target_module as local rules. If either source_module or target_module is not supplied, it refers to the global module. source_rules specifies which rules from the source_module to import; TARGET_RULES specifies the names to give those rules in target_module. If source_rules contains a name which doesn't correspond to a rule in source_module, or if it contains a different number of items than target_rules, an error is issued. For example,

# import m1.rule1 into m2 as local rule m1-rule1.
IMPORT m1 : rule1 : m2 : m1-rule1 ;
# import all non-local rules from m1 into m2
IMPORT m1 : [ RULENAMES m1 ] : m2 : [ RULENAMES m1 ] ;

The EXPORT Rule

EXPORT allows rule name aliasing across modules:

rule EXPORT ( module ? : rules * )

The EXPORT rule marks rules from the source_module as non-local (and thus exportable). If an element of rules does not name a rule in module, an error is issued. For example,

module X {
  local rule r { ECHO X.r ; }
}
IMPORT X : r : : r ; # error - r is local in X
EXPORT X : r ;
IMPORT X : r : : r ; # OK.

The CALLER_MODULE Rule

rule CALLER_MODULE ( levels ? )

CALLER_MODULE returns the name of the module scope enclosing the call to its caller (if levels is supplied, it is interpreted as an integer number of additional levels of call stack to traverse to locate the module). If the scope belongs to the global module, or if no such module exists, returns the empty list. For example, the following prints "{Y} {X}":

module X {
    rule get-caller { return [ CALLER_MODULE ] ; }
    rule get-caller's-caller { return [ CALLER_MODULE 1 ] ; }
    rule call-Y { return Y.call-X2 ; }
}
module Y {
    rule call-X { return X.get-caller ; }
    rule call-X2 { return X.get-caller's-caller ; }
}
callers = [ X.get-caller ] [ Y.call-X ] [ X.call-Y ] ;
ECHO {$(callers)} ;

The DELETE_MODULE Rule

rule DELETE_MODULE ( module ? )

DELETE_MODULE removes all of the variable bindings and otherwise-unreferenced rules from the given module (or the global module, if no module is supplied), and returns their memory to the system. Note: though it won't affect rules that are currently executing until they complete, DELETE_MODULE should be used with extreme care because it will wipe out any others and all variable (including locals in that module) immediately. Because of the way dynamic binding works, variables which are shadowed by locals will not be destroyed, so the results can be really unpredictable.

Local For Loop Variables

Boost Jam allows you to declare a local for loop control variable right in the loop:

x = 1 2 3 ;
y = 4 5 6 ;
for local y in $(x)
{
    ECHO $(y) ; # prints "1", "2", or "3"
}
ECHO $(y) ;     # prints "4 5 6"

Negative Indexing

Classic Jam supplies 1-based list indexing, and slicing on a closed (inclusive) range:

x = 1 2 3 4 5 ;
ECHO $(x[3]) ;   # prints "3"
ECHO $(x[2-4]) ; # prints "2 3 4"
ECHO $(x[2-]) ;  # prints "2 3 4 5"

Boost Jam adds Python-style negative indexing to access locations relative to the end of the list.

ECHO $(x[-1]) $(x[-3]) ; # prints "5 3"
ECHO $(x[-3--1]) ;       # prints "3 4 5"
ECHO $(x[-3-4]) ;        # prints "3 4"
ECHO $(x[2--2]) ;        # prints "2 3 4"       

Consistency with the 1-based, inclusive indexing of Classic Jam and the use of ``-'' as the range separator make this feature a bit clumsier than it would otherwise need to be, but it does work.

Support for Cygwin

When invoking Windows-based tools from Cygwin it can be important to pass them true windows-style paths. Boost.Jam supplies the :W modifier which, under Cygwin only, turns a cygwin path into a Win32 path using the cygwin_conv_to_win32_path function. On other platforms, the string is unchanged.

x = /cygdrive/c/Program Files/Borland ;
ECHO $(x:W) ; # prints "c:\Program Files\Borland" on Cygwin

Target Binding Detection

Whenever a target is bound to a location in the filesystem, Boost Jam will look for a variable called BINDRULE (first ``on'' the target being bound, then in the global module). If non-empty, $(BINDRULE[1]) names a rule which is called with the name of the target and the path it is being bound to. The signature of the rule named by $(BINDRULE[1]) should match the following:

rule bind-rule ( target : path )

This facility is useful for correct header file scanning, since many compilers will search for #included files first in the directory containing the file doing the #include directive. $(BINDRULE) can be used to make a record of that directory.

Return Code Inversion

For handling targets whose build actions are expected to fail (e.g. when testing that assertions or compile-time type checkin work properly), Boost Jam supplies a FAIL_EXPECTED rule in the same style as NOCARE, et. al. During target updating, the return code of the build actions for arguments to FAIL_EXPECTED is inverted: if it fails, building of dependent targets continues as though it succeeded. If it succeeds, dependent targets are skipped.

Ignoring Return Codes

Perforce Jam supplied a NOCARE rule which is typically used for header files to indicate that if they are not found, the dependent targets should be built anyway. Boost Jam extends NOCARE to apply to targets with build actions: if their build actions exit with a nonzero return code, dependent targets will still be built.

Removing Outdated Targets

rule RMOLD ( targets * )

Perforce Jam removes any target files that may exist on disk when the rule used to build those targets fails. However, targets whose dependencies fail to build are not removed by default. The RMOLD rule causes its arguments to be removed if any of their dependencies fail to build.

The SUBST Rule

Note: the SUBST rule is deprecated in favor of Perforce Jam's built-in MATCH rule, which has been rolled into Boost.Jam.

The behavior of the SUBST rule for regular-expression matching and replacement (originally added in FTJam) has been modified:

• One or more replacement patterns may be supplied. The new signature for SUBST is:

  SUBST ( source pattern replacements + )
  

  The return value is the concatenated results of applying each element of replacements in turn. For example, the following will print ``[x] (y) {z}'':

  ECHO [ SUBST xyz (.)(.)(.) [$1] ($2) {$3} ] ;
  

• If there is no match, SUBST now returns an empty list. In FTJam, the original source string was returned, making it awkward to check whether a pattern was matched.
• Compiled regular expressions are now internally cached, making it much faster to use SUBST multiple times with the same string.

The JAM_VERSION global variable

A predefined global variable with two elements indicates the version number of Boost Jam. Boost Jam versions start at "03" "00". Earlier versions of Jam do not automatically define JAM_VERSION.

Debugging Support

The BACKTRACE rule

rule BACKTRACE ( )

Returns a list of quadruples: filename line module rulename..., describing each shallower level of the call stack. This rule can be used to generate useful diagnostic messages from Jam rules.

The -d command-line option admits new arguments:

• -d+10 - enables profiling of rule invocations. When Jam exits, it dumps all rules invoked, their gross and net times in platform-dependent units, and the number of times the rule was invoked.
• -d+11 - enables parser debugging, if Jam has been compiled with the "--debug" option to the parser generator named by $(YACC).
• -d+12 - enables dependency graph output . This feature was ``stolen'' from a version of Jam modified by Craig McPheeters.

The UPDATE rule and changes to command line handling

Classic jam treats any non-option element of command line as a name of target to be updated. This prevented more sophisticated handling of command line. This is now enabled again but with additional changes to the UPDATE> rule to allow for the flexibility of changing the list of targets to update. The UPDATE builtin rule is:

rule UPDATE ( targets * )

The rule has two effects: 1. it clears the list of targets to update, and 2. causes the specified targets to be updated. If no target was specified with the UPDATE rule, no targets will be updated. To support changing of the update list in more usefull ways, the rule also returns the targets previously in the update list. This makes it possible to add targets as such:

local previous-updates = [ UPDATE ] ;
UPDATE $(previous-updates) a-new-target ;

Semaphores

It is sometimes desirable to disallow parallel execution of some actions. For example:

• Old versions of yacc use files with fixed names. So, running two yacc actions is dangerous.
• One might want to perform parallel compiling, but not do parallel linking, because linking is i/o bound and only gets slower.

Any target can be assigned a semaphore, by setting a variable called SEMAPHORE on that target. The value of the variable is the semaphore name. It must be different from names of any declared target, but is arbitrary otherwise.

The semantic of semaphores is that in a group of targets which have the same semaphore, only one can be updated at the moment, regardless of "-j" option.

The W32_GETREG rule

    rule W32_GETREG ( path : data ? )

Defined only for win32 platform. It reads the registry of Windows. 'path' is the location of the information, and 'data' is the name of the value which we want to get. If 'data' is omitted, the default value of 'path' will be returned. The 'path' value must conform to MS key path format and must be prefixed with one of the predefined root keys. As usual,

• 'HKLM' is equivalent to 'HKEY_LOCAL_MACHINE'.
• 'HKCU' is equivalent to 'HKEY_CURRENT_USER'.
• 'HKCR' is equivalent to 'HKEY_CLASSES_ROOT'.

Other predefined root keys are not supported.

Currently supported data types : 'REG_DWORD', 'REG_SZ', 'REG_EXPAND_SZ', 'REG_MULTI_SZ'. The data with 'REG_DWORD' type will be turned into a string, 'REG_MULTI_SZ' into a list of strings, and for those with 'REG_EXPAND_SZ' type environment variables in it will be replaced with their defined values. The data with 'REG_SZ' type and other unsupported types will be put into a string without modification. If it can't receive the value of the data, it just return an empty list. For example,

    local PSDK-location =
    [ PROFILE HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\MicrosoftSDK\\Directories : "Install Dir" ] ;

Jam Fundamentals

This section is derived from the official Jam documentation and from my experience using it and reading the Jambase rules. I repeat the information here mostly because it is essential to understanding and using Jam, but is not consolidated in a single place. Some of it is missing from the official documentation altogether. I hope it will be useful to anyone wishing to become familiar with Jam and the Boost build system.

· Jam ``rules'' are actually simple procedural entities. Think of them as functions. Arguments are separated by colons.

· A Jam target is an abstract entity identified by an arbitrary string. The build-in DEPENDS rule creates a link in the dependency graph between the named targets.

· Note that the documentation for the built-in INCLUDES rule is incorrect: INCLUDES targets1 : targets2 causes everything that depends on a member of targets1 to depend on all members of targets2. It does this in an odd way, by tacking targets2 onto a special tail section in the dependency list of everything in targets1. It seems to be OK to create circular dependencies this way; in fact, it appears to be the ``right thing to do'' when a single build action produces both targets1 and targets2.

· When a rule is invoked, if there are actions declared with the same name as the rule, the actions are added to the updating actions for the target identified by the rule's first argument. It is actually possible to invoke an undeclared rule if corresponding actions are declared: the rule is treated as empty.

· Targets (other than NOTFILE targets) are associated with paths in the file system through a process called binding. Binding is a process of searching for a file with the same name as the target (sans grist), based on the settings of the target-specific SEARCH and LOCATE variables.

· In addition to local and global variables, jam allows you to set a variable on a target. Target-specific variable values can usually not be read, and take effect only in the following contexts:

• In updating actions, variable values are first looked up on the target named by the first argument (the target being updated). Because Jam builds its entire dependency tree before executing actions, Jam rules make target-specific variable settings as a way of supplying parameters to the corresponding actions.
• Binding is controlled entirely by the target-specific setting of the SEARCH and LOCATE variables, as described here.
• In the special rule used for header file scanning, variable values are first looked up on the target named by the rule's first argument (the source file being scanned).

· The ``bound value'' of a variable is the path associated with the target named by the variable. In build actions, the first two arguments are automatically replaced with their bound values. Target-specific variables can be selectively replaced by their bound values using the bind action modifier.

· Note that the term ``binding'' as used in the Jam documentation indicates a phase of processing that includes three sub-phases: binding (yes!), update determination, and header file scanning. The repetition of the term ``binding'' can lead to some confusion. In particular, the Modifying Binding section in the Jam documentation should probably be titled ``Modifying Update Determination''.

· ``Grist'' is just a string prefix of the form <characters>. It is used in Jam to create unique target names based on simpler names. For example, the file name ``test.exe'' may be used by targets in separate subprojects, or for the debug and release variants of the ``same'' abstract target. Each distinct target bound to a file called ``test.exe'' has its own unique grist prefix. The Boost build system also takes full advantage of Jam's ability to divide strings on grist boundaries, sometimes concatenating multiple gristed elements at the beginning of a string. Grist is used instead of identifying targets with absolute paths for two reasons:

1. The location of targets cannot always be derived solely from what the user puts in a Jamfile, but sometimes depends also on the binding process. Some mechanism to distinctly identify targets with the same name is still needed.
2. Grist allows us to use a uniform abstract identifier for each built target, regardless of target file location (as allowed by setting ALL_LOCATE_TARGET.

When grist is extracted from a name with $(var:G), the result includes the leading and trailing angle brackets. When grist is added to a name with $(var:G=expr), existing grist is first stripped. Then, if expr is non-empty, leading <s and trailing >s are added if necessary to form an expression of the form <expr2>; <expr2> is then prepended.

· When Jam is invoked it imports all environment variable settings into corresponding Jam variables, followed by all command-line (-s...) variable settings. Variables whose name ends in PATH, Path, or path are split into string lists on OS-specific path-list separator boundaries (e.g. ":" for UNIX and ";" for Windows). All other variables are split on space (" ") boundaries. Boost Jam modifies that behavior by allowing variables to be quoted.

· A variable whose value is an empty list or which consists entirely of empty strings has a negative logical value. Thus, for example, code like the following allows a sensible non-empty default which can easily be overridden by the user:

MESSAGE ?= starting jam... ;
if $(MESSAGE) { ECHO The message is: $(MESSAGE) ; }

If the user wants a specific message, he invokes jam with "-sMESSAGE=message text". If he wants no message, he invokes jam with -sMESSAGE= and nothing at all is printed.

· The parsing of command line options in Jam can be rather unintuitive, with regards to how other Unix programs accept options. There are two variants accepted as valid for an option:

1. -xvalue, and
2. -x value.

Please also read The Jam language reference for the additional details.

Revised 18 November, 2003

Copyright 2003-2004 Rene Rivera, David Abrahams, Vladimir Prus.

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)