Automated Builds

Making new builds of a game in development is a very stressful task. It requires great care and concentration for several hours, on a task which is essentially tedious. Particularly towards the end of a project, the slightest mistake could be disastrous. Because of the demands of marketing, testing, management and the publisher, it is also often carried out under high pressure. Computers are particularly good at automating repetitive tasks, and carrying them out quicker and more reliably than humans can. Unfortunately, because this is often hard to do in Windows, people find it easier to carry on performing tasks manually using the GUI. Hopefully this situation will be better with the improved scripting features in Windows 2000, but until then we have to use other means. In Creatures 3, we used Unix scripting tools to automate the build process. This made the following improvements: • • • More reliable. The final builds were of a higher quality than manually made builds. This is because the build script never forgets to do something. In addition, automatic testing during the build process ensures the most obvious errors are caught before the build is finished. Save developer time. Instead of having one engineer working full time to make a build, you only need someone to start the build going, and organise fixing of any errors that show up. That one engineer can then fix bugs, or do something else useful. Risk reducing. If the developer who normally makes builds is ill, or otherwise unavailable, it is easier for other people to make builds. Even if there are problems, there is a clear script describing every stage of the build, so it is possible to work out what went wrong. The knowledge in the mind of the developer is captured in the scripts, rather than in his head. Faster. If things go smoothly, a new build can be made in half an hour. This isn’t hugely faster than a manual build when things go smoothly. The time saved is in gaps between stages of a build – the computer just carries straight on with the next phase – and that the build can be made overnight, or while at lunch. Instant availability for testing. The build process empowers QA by emailing them to let them know as soon as a new build is available. Everyone knows straight away when a new build is ready.

Despite all these improvements, making builds is still slightly stressful! The person responsible for builds still needs to check that everyone in the team is ready for the new build to be made, start the build going, check for any errors and fix them or arrange for them to be fixed, and maintain the build scripts. Right until the last week of the project sections were added to the Creatures 3 build script to check for more errors, and afterwards more changes were made for localisation. The Creatures 3 build process ran on a dedicated machine, which was of high specification at the time. This stops it interfering with the developer’s main machine, and is also useful for soak tests, and other tests running in the debugger.

Creatures 3’s Build Process
Creatures 3’s build process uses Unix tools running under Windows. These are free tools, licensed under the GPL, and ported to Windows by Cygnus (www.cygnus.com). The main tool is the Bash (Born Again SHell), which is a standard Unix command line shell. It has a powerful scripting language, which provides all the logic conditions and loops that you need, and much more. The standard Unix file commands mv, ls, cp, mkdir, rm, and cd are used to move files on and off the network and between directories. Egrep is used to help perform various checks on files, using its powerful

regular expressions. Sed (or Perl) easily search and replace inside text files. Diff is also used to perform various checks. Error checking is done using the error codes which command line programs return to say whether they were successful or not. Bash lets you relatively easily trap all errors – for example aliases can be set up so that cp (the file copy command) and other command commands always have error checking, and abort the script if there is a problem. If there is an error, the build script automatically emails the developer responsible for builds with a log of what happened. If the build completes successfully, it emails the appropriate mailing lists or newsgroups via the Exchange server. This is done by a command line program Smtpmail which sends email by SMTP. The IT department can alias email addresses to mailing lists or newsgroups by configuring Exchange. The build is broken into various sections, described below. The script’s command line parameters let the user run all the sections at once for an unattended build. If there are problems or the build needs restarting in the middle after correcting an error, the sections can be run one at a time or in groups. Often we wouldn’t run a section of the build – for example, the engine doesn’t always need recompiling.

Engine
Towards the end of a project, you often need to make several new Release versions of the game engine in one day. Even though the process isn’t that complicated, it still proved worth automating – if only to free a developers main machine and mind so they can carry on debugging. This part of the script makes a new version of the main game engine. It checks out the header file containing the version number, and increments it, then labels all the code in SourceSafe with that version. Then the code is fetched from Sourcesafe, compiled using msdev, and the finished executable copied to a standard place on the network. So that everyone knows what has changed in the new executable, there is also a ChangesSinceLastVersion.txt file in Sourcesafe. Whenever someone makes a change, they add a comment to this file saying what they have done. The engine build process checks this file out, and posts its contents to an internal newsgroup announcing the new engine. It then clears out the file for future use, and checks it back in. At last, agent engineers know exactly what is new! As this part of the script uses source safe, extra care is needed for error recovery. The script undoes checkout of any files it might have checked out before it begins, so any previous failed runs are cleared up. It is also careful about the order that things are done in, so the files labelled are actually those that are compiled, and so that developers can’t change key files during the build.

Prepare
This part of the script makes an image of the game on the local hard disk, by gathering data from various standard places where the development team have put it. It copies image and sound files from a build template directory, CAOS script and catalogue files from source control, splices new genomes together, and compiles PRAY files (equivalent of COBs, some are used in the main game). A new build number is generated. This is done by looking on the Completed Builds directory on the network, and finding out what build numbers are already used. A new number is generated one higher than

the largest existing one. The catalogue files are patched so the build number appears in the about box in the game. It also removes debugging lines from the scripts, and removes the comments lines so users can’t see them.

Language
After localisation, this stage sets up the appropriate language. There are seven types – UK, US, French, German, Dutch, Spanish and Italian. According to the command line parameters, this adjusts the build image to work with that language, removing some excess data which is only for the other languages. This includes using Sed to edit the InstallShield configuration files, so when the build is compiled it uses the correct language. It also handles special localisation tasks, such as adding the AOL installer to the US build only.

Check
This runs various checks to make sure nobody has messed up. For example, it scans the scripts for all lines which generate a sound, and checks the sound files are available. These checks were added as and when a problem came up which could be automatically detected, so that the same problem couldn’t happen again.

Test
Now the game image is ready on the build machine. The test section of the script actually copies the image in to the Program Files/Creatures 3 directory, and launches the game. Various CAOS scripts are added to this copy of the build, so that the world switcher is automatically bypassed and the main world loaded in. The Bash script then waits for ten minutes, checking for errors produced by the engine every minute. If there are any errors, it aborts the build process. The engine automatically writes all error messages to a log file, so the build script simply needs to check for presence of this file, and post the contents in an error email. Then the script kills the engine process so it has all the resources of the build machine to carry on with the next stage. The finished build is now guaranteed to run unattended for ten minutes without any error (excepting due to different random number sequences!). In addition, for Creatures 3 the starter family is also made during this test phase. The two parent Norns are created by a CAOS script which is injected into the game by the build script. The Norns are aged, taught all vocabulary, and allowed to walk round and play for a while, and also encouraged to be friendly with each other so they will breed more easily on the users’ machines. After this quick introduction to life, they are exported. The build script then copies them from the Program Files directory back to the build image. The CAOS script does some extra checks at this stage. For example, it ensures that the female starter Norn doesn’t actually get pregnant!

Compile
Creatures 3 uses InstallShield for installation. This section of the build script doctors some of the configuration files, compiles the InstallShield scripts, and builds an installation. It then copies that and makes a CD Image on the build machine.

InstallShield’s compression features aren’t used, so the files are uncompressed on the CD. This makes it easy for people to check what was in a particular build, and also for users to retrieve corrupt files without a complete reinstallation. A manual installation is also possible if something is wrong with InstallShield. This CD Image is now complete, and can be manually tested on the build machine, or installed directly from there to other machines.

Commit
The final stage of the build is to copy the CD Image to the network. It copies it to a folder called “new build still being copied” so nobody tries to install the build while it is still being transferred. When the copy is complete, it renames it to the appropriate name and number, including the language. For example, “build 98 spanish”. The builds are all put in a standard “completed builds” directory on the network. This makes it very easy for people to find them. In addition to copying the CD Image, the InstallShield project and scripts are copied to back them up. It would be possible to copy all source code here as well, so the complete source of each build is available for inspection. When done, an email is sent to all developers and testers, telling them the new build is available.

Wolf
This final section is mainly for overnight builds. It launches the game, and makes 8 brand new Norn eggs. The game is then put into wolfing mode, turning off the graphics update so it runs faster. All errors are logged to a file, and the game is left running. This proved very useful for catching lots of common errors as early on in development as possible. It was also possible to use the wolfing run to report bugs in ecosystem stability, and Norn longevity or breeding. Because the wolfing happens automatically at the end of the build process, it is a more reliable way of ensuring the team actually does this than people trying to remember to do it on their own machine before they go home.

Problems
There were several problems or frustrations with the Creatures 3 build process, or the tools used. • Bash is unstable with respect to launching processes. This is because Unix uses the fork command, which duplicates a process in memory, and Windows has no direct counterpart. Cygnus (who ported the tools to Windows) have mostly overcome this problem, but sometimes bugs do show up. The software is still in beta. The problem always manifested itself (with me) when I had too many commands in one section of script. The solution was to break that part of the script into a separate file, and launch a new instance of bash to run it. This was quite annoying and inconvenient. Incompatibility with file paths. Because the build process uses Unix tools, the tools expect a Unix format file system. This has everything mounted under one root node /, rather than having drives c:, d:. All of the tools can happily access files using paths like “/c/program files/creatures 3/engine.exe”, or network paths “//cyberbackup/clabs xtra/creatures 3/”, the problem comes when calling external Windows programs. Usually, it is OK if you are in the same current directory as you want to work in. Sometimes, however, the same path had to be stored twice, once in Unix form to pass to a Unix tool, and once in Windows form to pass to a Windows designed program.

• • •

Spaces in filenames. This is an inconvenience rather than a problem. Because of spaces in filenames, the syntax of cp and mv and other commands can be counterintuitive. This took up more time than expected, and the only solution seems to be to try to avoid spaces, and to learn this aspect of Bash syntax carefully. Slower than it should be. The build process sometimes performs excessive file copying. For example, it copies the CD Image, including intro sequences, to the build machine, and then back to the network. With careful thought, a build process could be quicker. Bash slower than it should be. The Unix tools are slower under Windows than if run under Unix on the same machine. Takes time to write scripts. If using the same system in future, this would be a lot quicker as some existing sections of script could be used. However, in any future project the scripts would need to be rewritten and maintained.

Ways to solve some of these problems are discussed in the next section.

Future Ideas
Alternatives to Unix Tools
Because of the problems with fork and path names described above, and general slowness, it is worth considering using a tool other than the Cygnus Unix tools. The following are alternatives: • DOS command line. The standard Windows command line DOS prompt would be able to do some of the automation described above, but not all of it. It would be particularly bad at automated checking, and be incapable of altering InstallShield configuration files. This is a possibility in combination with Perl (see below). Take Command from JP Software (www.jpsoft.com) offers command line scripting which is backwards compatible with DOS, and is as powerful as Bash. However, this wouldn’t offer Unix tools - no sed, perl or grep. As with the DOS command line, it could be used with Perl. It is also commercial, but not very expensive to buy for just one machine. Perl is an excellent language for lots of things, and has a reliable Windows port. It is still a Unix tool ported to Windows, and I don’t know how much it improves over Cygnus Bash when it comes to forking, filenames and speed. It isn’t ideal for copying files, however, so might be best used with a command line tool. Windows 2000 Scripting Host. This is an unknown, as Windows 2000 isn’t released yet. And we still have to be compatible with Windows 95. Another port of Bash. It might be worth seeing if there is another, better, non-Cygnus port of Bash. This seems unlikely, however.

• •

Cygnus (www.cygnus.com) have been recently purchased by RedHat, the most famous Linux distribution company. It remains to be seen whether this means they maintain Cygwin (the Windows ports of Unix tools) better or worse, although their web site says “we anticipate no change in the products and services offered by either company”. Certainly, progress with Cygwin development hasn’t been quick over the last couple of years, but it has been steady. It would only take an inspired bug fix to make Bash much more useful. We could even try fixing it ourselves, as the source code is freely available. There are two other advantages in using standard Unix tools. The skills learnt are transferable – possibly important for using the Playstation 2 development kit to its full (it has a Linux core), or if we ever make Linux versions of games. Secondly, there is already experience in the company, both of the specific issues for a build process, and in general of using Unix.

Alternatives to InstallShield
Although there were no fatal problems with InstallShield, it often felt like it was more of a burden than a help. Here are some of the issues. • There are many advantages of not compressing files on the CD, which InstallShield would happily do. One advantage is that you could make new builds with small, controlled changes, by cloning old builds, and updating just the appropriate files. However, if you patch such a build by editing a file, InstallShield truncates that file during installation. Presumably it doesn’t perform a simple file copy on uncompressed files, but still runs it through some of its compression code pipeline, with decompression turned off. This is very unhelpful. Cumbersome to program from the command line. It really is designed for GUI work. To use from the command line, its configuration files have to be patched each time (for different directories and languages), and files need copying round to cope with the places InstallShield likes to put things. Localisation was different. Because it wasn’t our own program, it had to be localised in a different way. We couldn’t use the catalogue file system.

• •

In addition, there would be the following possible benefits from using something other than InstallShield. • • Potentially prettier, and more flexible interface. Make it feel more like a game, by launching straight into our own sequence from the moment the CD is put into the drive. InstallShield makes everything look like a boring business application. Possibility of running the build process on Linux, or at least important parts of it. Since Unix is better designed for command line, it is well suited to this task, and this would solve many of the problems described above. This is especially possible now we are using CVS. Unfortunately, there are still other tools used – Msdev for engine builds, and running the game itself during automated testing – which are only available under Windows. It’s worth considering running parts of the build as scripts on the Linux CVS server for the game – if only to reduce network traffic.

The following are alternatives to using InstallShield. • Hand rolled installer. There are surprisingly few things which InstallShield actually does which are helpful. Most of its code is in its own script language, and has been tweaked by us anyway. This could just as easily be C or C++ code. InstallShield’s scripting language is very primitive, needs to be learnt, and has an inferior debugger to Msdev. It is surprisingly similar to C in function and purpose. DirectX installation, disk space checking, and setting up the registry were all handled by our own custom script code in Creatures Adventures and Creatures 3. This could as easily be done in C. File copying would be better done in C, as it avoids the truncation problem described above. Similarly, the display and dialogs could have a prettier and more “fun” feel. You can change the dialogs in InstallShield, but it seems more trouble than it is worth to try and do so in the scripting language. There are two things which InstallShield does which are less obviously easy to duplicate. These are installing DLLs, and Uninstallation. For CD published games, all DLLs can be statically linked, or included in the same directory as the executable. Uninstallation can relatively straight-forwardly delete the entire directory – the hook to the add/remove dialog in control panel should be fairly easy to do. Uninstallation also would need to include rollback for failed installs. We used essentially hand-rolled installers for patches of both Creatures 2 and Creatures 3. These already include some of the code above. They also effectively roll the Autorun code into the installer code – checking for things like the game already running, or already being installed.

Other products. There are other installation products, and we could review them. One (can’t remember which!) simply generates C code from an initial Wizard, which might be helpful. However, in general, I suspect that the other install programs won’t be better than InstallShield. When reviewing alternative Windows Help and HTML Help compilers at a previous company, I found that the competition was as bad as the one that I was already using. I suspect that this will be similar for installation.

It’s worth considering Windows 2000 again here. It has new installation features (the Windows Installer), which could (should! in a good operating system) make it much easier for us to hand roll an installer. Unfortunately, we have to be compatible with both Windows 2000 and Windows 95. InstallShield has a new 2000 version which works with the Windows Installer. It’s not clear why this is necessary, providing Microsoft have done a good job. Either they haven’t, or InstallShield’s Windows 2000 installer only gives minor advantages over using Microsoft’s directly.

Structure of the build process
There are several ways in which the Creatures 3 build scripts could be improved. • Make sure you minimise time to make a build. Sometimes files are copied in an excessive way. For example the whole CD image is copied to the local machine and back to the network. Parts such as the introduction sequence could be copied directly between places on the network. As circumstances vary here, the only lesson to bare in mind is that copying large amounts of data across the network, or between two places on one machine, is very slow, and it is worth taking the time to reduce this. Quicker builds, and reduced server strain are the benefits. Useful if it could automatically restart in the middle. The Creatures 3 build process doesn’t have any sense of where it is up to. Sometimes you can manually restart from halfway through, but you have to know what you are doing. It would be useful if the script tracked this. That way, when an error is found, the build can be resumed as quickly and reliable as possible, without repeating complete stages. What’s the best way to segment the build into sections? The scripts are broken into eight sections: engine, prepare, language, check, test, compile, commit, wolf. These sections can be tested and run independently. It would be useful to further break things down, as often there are independent subsections within each section. This would work well with the “restarting in the middle” described above. Interface available to the whole team. It might be useful to have a web interface displaying progress of the current build, and allowing anyone with authority to start a build. This would make the build process a tool owned by the whole team, rather than something one individual has to control. The benefits that you can transfer responsibility for completing a build to a person other than the person who maintains the builds scripts, that people can easily make builds when that person is unavailable, and that people always can find out what is going on. It might not be worth doing, as it is often helpful to have one person co-ordinating things. Automatic error reporting. It would be possible to categorise errors, and report them automatically to the appropriate person, by email. This works well with the web interface, and has similar advantages and disadvantages. Things worth considering are the ability to run multiple builds at once (different language versions, for example – obviously, you need either several computers, or for the build process to not totally overload one machine), or alternatively to easily sequence several builds.

Modular and general purpose. It might be possible to make the build scripts more general purpose, so they can be used on several projects. It’s not clear that this is practical, as every project will be quite different in its requirements. Scavenging code from previous projects might be easier. One thing to do would be to separate key bits into separate scripts – for example, “compiling with InstallShield”, and “getting a project from source control and compiling it with Msdev”. This library of scripts could then be shared between projects.

Francis Irving November 19th 1999