SCons: an introduction

Dean Giberson <dean@deangiberson.com> Dec 17, 2008

If you've tried putting a build system together for a modern game you know that you are facing a monumental, never ending task. Once you do get a basic pipeline working, modern games require source assets that are in the range of 100 Gb of raw data. Tracking this amount of data takes time and resources. SCons (http://www.scons.org) is a Python based make replacement; with it you can tame your dependencies and data sizes. What follows is quick art centric introduction.

1 The Canonical Example
The SCons documentation1 would give an example like this:
env = Environment() env.Program( 'HelloWorld', ['HelloWorld.cpp'])

follow this and provide a compilable HelloWorld.cpp and you would have a program ready to go. But how would you extend this to use your own pipeline tools?

2 Adding Compressed Textures
Lets assume that you want to compress textures for use in a game. NVidia Textures2 Tools can be used on your TGA/PNG/JPG/DDS images. For example you want to convert this image3 into a DXT5 compressed image for use on target:
nvcompress -color -bc3 gunmap.png gunmap.dds
1 http://www.scons.org/doc/0.97/HTML/scons-user/book1.html 2 http://developer.nvidia.com/object/texture_tools.html 3 Images provide by G3D Engine Data Collection http://g3d-cpp.sourceforge.net/.

1

png') 2 .dds') env['BUILDERS']['NvCompress'] = NvCompressBuilder env.CLVar('-color') env['NVCOMPRESSFLAGS'] = SCons. import SCons env = Environment(ENV = os.Once we understand how to do this from the command line adding it to SCons is quite easy.NvCompress( 'image'.Util.Util.environ) env['NVCOMPRESS'] = 'nvcompress' env['NVCOMPRESSTYPE'] = SCons.CLVar('-bc3') env['NVCOMPRESSCOM'] = '$NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET' NvCompressAction = Action( '$NVCOMPRESSCOM') NvCompressBuilder = Builder( action=NvCompressAction. suffix='. 'image.

Normally you don't have to do this.3 A Quick walk-through to understand what is happening import SCons We import the SCons python module into our script. SCons does not copy the system environment by default. We can add to them through assignment. but please note that this is bad practice. • a template command line ('NVCOMPRESSCOM').CLVar('-color') env['NVCOMPRESSFLAGS'] = SCons. -color -bc3 $SOURCE $TARGET' 3 'nvcompress . env['NVCOMPRESS'] = 'nvcompress' env['NVCOMPRESSTYPE'] = SCons. • the default type ag ('NVCOMPRESSTYPE'). For now I'll just copy the system environment. SCons uses late binding of variables found within environment strings. Environment objects are class instances that behave like dictionaries.Util. I'll come back to what this means in practical terms in a moment.Util.Util. • the default compression ag('NVCOMPRESSFLAGS'). this is by design as a build environment should be as explicit as possible. and query the contents using normal Python dictionary functions. but I'm using the 'SCons.environ) Construct a SCons Environment object. any sub-string that starts with a '$' character is interpreted at the calling site for the current value within the current environment.CLVar' class provided by SCons so I import the module to gain access to it. In our case I'm lling four slots with: • the name of the executable ('NVCOMPRESS'). env = Environment(ENV = os.CLVar('-bc3') env['NVCOMPRESSCOM'] = '$NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET' Now begin populating the Environment with data. but for now accept that our default command line evaluates to: . This acts like a controlled dynamic scoping system.

NvCompressAction = Action( '$NVCOMPRESSCOM') NvCompressBuilder = Builder( action=NvCompressAction. Always in this order. We also set the default extension for all targets. an Action object is created. the action is called. After control returns to the SCons system the Step 3 begins and the dependency tree is scanned and the actions are triggered. 'image. Work is done later by an internal SCons class. 3. and a name is given for calls to it. It doesn't actually run 'nvcompress' at this point.NvCompress( 'image'. The dependencies are scanned. Create a tree of dependencies with sources and targets.dds') These three lines are the core of our SCons extension. then for any out of date data. Step 2 happens within the SConstruct script and continues until the end of the script. and must be occur in this order. It's for this reason that I say that a dependency node is constructed from the last line of the SConstruct. Only an object. 4 . The evaluation of this command follows the same rules as for environment variables set earlier. 2. Then a Builder is constructed using the Action object just created. suffix='. is constructed and added to the system. Each of these steps is discrete. 'Taskmaster. Step 1 happens during the setup phase of SCons (default tools are scanned for and basic environments are constructed) and to a lesser extent within the running SConstruct script itself (as I've just shown). SCons is a three stage process: 1.png') The only thing left is to construct a Node in the dependency tree for the target le. Create or gather a collection of tools and set the environment. and the command line string is set. env. representing the potential to run 'nvcompress'. env['BUILDERS']['NvCompress'] = NvCompressBuilder The Environment is then extended with this builder.

NvCompress( 'image2'. Any textures that are added are found automatically. have access to all of Pythons functions.dds'.'. Normally you will want a range of compression methods for textures. NVCOMPRESSFLAGS='-bc2') image2. A consequence of this choice is that you.png Adding a method to match lename patterns in a database (or text le) gives us a simple way to control the compression of individual textures./*/*.org/modindex. Glob from glob import glob for tex in glob( r'. SQL.png'): target = tex. env. 5 Other compression types The previous method of adding dependencies will add each texture into the pipeline with the same compression options. I mentioned earlier that the evaluation of strings set for Builders and Actions acts like dynamic scoping for variables. I've shown you how to add one Builder and access that Builder for a single target texture.dds') env.NvCompress( target. The same strategy can be applied to other types of data as well. to the rescue. and add them to the dependency graph. Check the module documentation4 for details.html 5 . more controls One of the great things about SCons is that it's embedded with Python. changing the values in the Environment. you're going to want access to many textures and that means several targets. You will not have to drop out of Python for very many reasons. Games are not made from one texture.png'.4 More les. Which will result in this command: 'nvcompress -color -bc2 image. tex) This will nd all of the source textures in subdirectories. the tool writer. I want to focus on the clearest method. 'image. even direct in memory structure access or peeking in compressed les.python.replace('. This feature allows us to change the functionality of a call by changing values when the dependency node is built. Python has great libraries for XML. # Global texture compression options 4 http://docs.png'.

a comma ('.dds') for pat. set the compression options for that texture.pat): opt = opt.\envmaps\*.') gCompressionOptions.-bc3n # Does not include cubemaps This simple text le has a line for each le pattern.txt'.png'.'.\nmaps\*. from glob import glob from fnmatch import fnmatch gCompressionOptions = [] f = open('texture_options.close() for tex in glob( r'.split('.opts' .-bc1 . 6 .options) = line. A parser for this format is easy to write.') and the compression option for the command line.png'): hasCustomPattern = False target = tex. 6 Exploring dependencies A core strength of SCons is it's dependency system. If not the default is used.options)) finally: f.# format 'glob.png.replace('. Comments start with a hash character ('#') and continue to the end of the line.NvCompress( target.png. tex) Once we have the patterns into an array it's simple to check if any les found by the glob matches a given pattern.strip() env. tex. This system is not normally based on time stamps but on a one way hash of the le contents (MD5).\*\*.append( (pattern.split('#')[0] if line != '': (pattern. Using hash values allows for stronger connections between assets.'rt') try: for line in f: line = line.opt in gCompressionOptions: if fnmatch(tex. NVCOMPRESSFLAGS=opt) hasCustomPattern = True if not hasCustomPattern: env. If there is a match.NvCompress( target.

dds +-nmaps | +-nmaps\gunmap.\nmaps\gunmap. The odd part of this result is that every person is trying to get the same result from the same source. This doesn't help on a large team.' is up to date.In order to follow what SCons is doing with dependency checking you can use the command line option. Then when you go to build. you must change the contents of the le or force SCons to rebuild an asset. scons: Building targets .5 seconds.dds scons: done building targets. special . SCons took the strong hash key info and took it to the next logical level. if you have 20 other people on your team. 7 Sharing the results Strong dependency systems are great for personal development. 'debug=explain' touch tree=derived scons: Reading SConscript files .dds . if an artist makes a change then once submitted every person on the team needs to build that same asset. On the ip side of this. you get control of the derived le from both the contents of the source les and the contents of the command line used to build the derived le. +-envmaps | +-envmaps\gunmap.png -bc1 special . To get a view of the dependency tree use the option.png -bc3n normal . +-.\nmaps\gunmap. do 7 .dds +-tex +-tex\gunmap....dds . then it's possible to store copies of the result in a shared location under that name.\envmaps\gunmap. If everyone calculates the hash the same way.\envmaps\gunmap. scons: `.. SCons combines all of these values into a string that represents the derived target asset. The interesting thing about using hash values for dependency check is that you can't use to force a recompile of an asset.dds . This option will print out information about dependency checks and commands being run. you can be sure that you only need to build the minimum following a change.\tex\gunmap. if any sources have changed that targets action is invoked.\tex\gunmap. In my case building a compressed texture takes 9. you team will spend 190 seconds for each texture change.png scons: done reading SConscript files.

The result is a distributed build cache.Util. Create a location with a lot of available disk space. env. 8 The script we have Taking all of these changes into account we get the following script.environ) env. If the asset exists under that name in this cache then just copy it. normally the originator of the source art.CacheDir(r'x:\Location\of\cache\dir') env['NVCOMPRESS'] = 'nvcompress' env['NVCOMPRESSTYPE'] = SCons. Next place this line into you SConstruct le. and only one person.dds') env['BUILDERS']['NvCompress'] = NvCompressBuilder from glob import glob from fnmatch import fnmatch gCompressionOptions = [] f = open('texture_options. needs to build the nal result.CacheDir(r'x:\Location\of\cache\dir') Now your derived les are shared.a quick check rst.Util. And you can take advantage of it out of the box. suffix='. In most cases.txt'.CLVar('-color') env['NVCOMPRESSFLAGS'] = SCons.'rt') try: 8 . import SCons env = Environment(ENV = os. this saves hours of cumulative time for a large team of people. available for everyone on your team. if not then build the asset and place it into the cache under it's hash name. You results will be better if you have a continuous integration server for art. some server.CLVar('-bc3') env['NVCOMPRESSCOM'] = '$NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET' NvCompressAction = Action( '$NVCOMPRESSCOM') NvCompressBuilder = Builder( action=NvCompressAction.

using the compression options found in the le.options) = line.dds') for pat.NvCompress( target. tex. I hope you see how easy it is to build a strong build system for art given the right tools. NVCOMPRESSFLAGS=opt) hasCustomPattern = True if not hasCustomPattern: env.pat): opt = opt. texture_options.append( (pattern. having a quick.') gCompressionOptions.strip() env.\*\*.options)) finally: f.png'. but can be used for art builds as well.replace('.split('#')[0] if line != '': (pattern.for line in f: line = line. and share the results with colleagues using a shared drive.close() for tex in glob( r'. robust pipeline is within your grasp. SCons is not just for code. tex) This will build all of your textures found in sub directories.png'): hasCustomPattern = False target = tex.split('.NvCompress( target. Coupled with a continuous integration server.'.opt in gCompressionOptions: if fnmatch(tex.txt 9 .

Sign up to vote on this title
UsefulNot useful