5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

Infrastructure & Automation

A deep dive into testing with TaskCat

by Shivansh Singh | on 25 JUN 2019 | in Advanced (300), AWS CloudFormation, How-To, Infrastructure & Automation | Permalink |
  Share

In the Up your AWS CloudFormation testing game using TaskCat blog post, I talked about how you can use TaskCat
to automate and expedite the testing of your Amazon Web Services (AWS) CloudFormation templates. In this post, I
will dive deep into the TaskCat features, various options available, and some tips and tricks for effectively using it to
test templates.

For this post, I assume that you already have TaskCat installed in your development
environment—whether locally on your laptop or in a cloud-based integrated
development environment (IDE) like AWS Cloud9. If you don’t have TaskCat
installed yet, you can refer to the TaskCat documentation or the Up your AWS
CloudFormation testing game using TaskCat blog post.

To get the best results when testing your templates with TaskCat, it’s helpful to
understand three primary files that TaskCat uses.

Parameter files
Standard file(Required)
Override files (Optional, but recommended for certain scenarios)
Configuration file (Required)

Let’s take a look at each of these files in more detail, and at how you can use them
to create and configure the test cases for your CloudFormation templates.

Parameter files

The parameter file is a JSON file that specifies test values for CloudFormation parameters that require user input.
When TaskCat launches CloudFormation templates for testing, it uses this parameter file to read the parameter
values and pass them to the CloudFormation template for stack creation.

You can have multiple parameter files to test different deploy

deployment scenarios for your stack. These parameter files
can live anywhere in your project folder. However, we recommend keeping them in the ci folder within your
project folder.

Structure

The parameter file must include a ParameterKey and a ParameterValue pair for each parameter in your
CloudFormation template that doesn’t have a default value.

[{
"ParameterKey": "KeyPairName",
"ParameterValue": "mykey"

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 1/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

}, {
"ParameterKey": "InstanceType",
"ParameterValue": "t2.small"
}]

You can also provide a ParameterKey and ParameterValue for a parameter if you want to override the default
value that’s in your CloudFormation template.

Tokens

Your CloudFormation template might have parameters that require values to be automatically generated—for
example, an alpha-numeric password, user name, etc. TaskCat has pre-defined tokens, which can be used as the
ParameterValue in the parameter file. These tokens needs to be specified as $[TOKEN-NAME]. During runtime,
TaskCat will generate the parameter value based on the specified token and pass it as a parameter value to the
CloudFormation template. Here are the currently supported tokens that generate values in TaskCat.

taskcat_genpass_N – Generates a random N character alpha-numeric string. N in the token name specifies the
length of the string. For example, $[taskcat_genpass_8] generates tI8zN3iX8.
taskcat_genpass_NS – Generates a random N character alpha-numeric string with special characters. N in the
token name specifies the length of the string and S specifies that the generated string should also contain
special characters. For example, $[taskcat_genpass_8S] generates mA5@cB5!.
taskcat_genaz_N – Generates comma-separated values of N Availability Zones. Availability Zone values are
generated based on the AWS Region where the stack is launched. N in the token name specifies the number of
Availability Zone values to be generated. For example, if the Region is us-east-1, $[taskcat_genaz_2]
generates us-east-1a, us-east-2b.
taskcat_autobucket – Generates a valid Amazon Simple Storage Service (Amazon S3) bucket name. For
example, taskcat-tag-sample-taskcat-project-5fba6597.
taskcat_genuuid – Generates a Universally Unique Identifier (UUID). For example, 1c2e3483-2c99-45bb-
801d-8af68a3b907b.
taskcat_random-string – Generates a random string of length 20 with alphabetic characters only. For
example, yysuawpwubvotiqgwjcu.
taskcat_random-numbers – Generates a random string of length 20 with numbers only. For example,
56188163597280820763.

In certain use cases, you may want to generate a random string for a parameter and then re-use that same
automatically generated value for another parameter of the template. An example is a parameter for the Database
password and another parameter to confirm the same Database password. You can generate a random
alphanumeric password by using the taskcat_genpass_N token and then refer its value by using another token
called $[taskcat_getval_<ParameterKey>]. The following example shows how an alphanumeric value of length 8 is
generated for the MyPassword parameter, and then its value is used for the ConfirmMyPassword parameter.

JSON
[{
"ParameterKey": "MyPassword",

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 2/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

"ParameterValue": "$[taskcat_genpass_8]"
}, {
"ParameterKey": "ConfirmMyPassword",
"ParameterValue": "$[taskcat_getval_MyPassword]"
}]

The following image shows an example parameter file (on the left) for a stack that requires user input for seven
parameters, and the automatically generated values (on the right) after runtime injection.

Override parameters

In certain situations, you might want to override the defined parameter values with your own values. For example,
you might want to override parameters such as KeyPairName or LicenseKey for your own environment. TaskCat
supports this functionality with two files:

Global override file – Place this in the .aws directory within your home directory:
~/.aws/taskcat_global_override.json
Per-project override file – Place this in the ci subfolder of your GitHub repository:
<project_name>/ci/taskcat_project_override.json

TaskCat will read parameters in the following order:

1. Read values from the standard parameter file.

2. Replace those with values from the global override file, if that file exists.
3. Replace those with values from the per-project override file, if that file exists.

Configuration file

Another important file required by TaskCat is the configuration file, which is called taskcat.yml in my project.
You can name it whatever you want, but you must have this file in your project folder. Though it can be anywhere in

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 3/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

your project folder, I recommend putting the configuration file in the ci folder within your project.

Here is the content of TaskCat configuration file for a sample project.

global:
owner: quickstart@amazon.com
qsname: test-project
regions:
- ap-northeast-1
- ap-northeast-2
- ap-south-1
- ap-southeast-1
- ap-southeast-2
- ca-central-1
- eu-central-1
- eu-west-1
- eu-west-2
- sa-east-1
- us-east-1
- us-east-2
- us-west-1
- us-west-2
reporting: true
tests:
test-scenario1:
parameter_input: input.json
template_file: master.template
regions:
- us-east-1
test-scenario2:
parameter_input: input2.json
template_file: master.template

The configuration file contains two high-level mappings: global and tests.

The global mapping defines the global configurations of the project. The global configuration contains:

owner – The project owner’s email address

qsname – The name of the project; it must be same as the project folder name
regions – All the AWS Regions where tests need to be executed
reporting – Whether to generate a test report with logs from each test execution

The tests mapping defines test scenarios that will be performed by TaskCat. You can define multiple test scenarios in
the tests mapping, and each test scenario must specify a parameter input file name and a CloudFormation template
file name.

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 4/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

It’s important to note that you only specify the name of the parameter and template files. TaskCat expects to find
parameter files and CloudFormation templates in the ci and templates folders, respectively, within the project
folder. You can also define the AWS Regions in which the test needs to be executed. This is optional. If you specify a
Region for a test scenario, this Region list will override the global Region list, and the test will be performed only in
the Regions defined for that test scenario.

Packaging AWS Lambda functions

If you are creating a Lambda function in your CloudFormation template and have its source code as part of your
source control repository, there is a good news for you. TaskCat can automatically zip your Lambda function source
and upload it to a temporary S3 bucket before launching your CloudFormation templates for testing.

To use this TaskCat feature, you need to put your Lambda function source in the
functions/source/<function name>/ folder of your code repository, and add package-lambda: true to the
global section of the TaskCat configuration file.

Now, when you run the TaskCat function, folders are zipped and placed in the following location:
functions/packages/<Function name>/lambda.zip .

Command line options

When using TaskCat to test your CloudFormation templates, you can use variety of command-line options to enable
or disable different features. These options help you further customize your test execution. You can see all the
available options by running the taskcat command:

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 5/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

However, I want to highlight few of the options that can come in really handy when you are testing your
CloudFormation templates.

Authentication

By default, TaskCat uses the default AWS credentials configured by you for AWS Command Line Interface (AWS CLI)
or boto3. However, you have two ways to override the default and use different credentials when running your tests.
This is useful when you work with more than one AWS account and have multiple profiles configured in your AWS
credentials file.

First, you can explicitly use a boto profile when running TaskCat. Use the -P option and pass a boto profile name
to use it for your test, as in this example.

Python
$ taskcat -c ci/taskcat.yml -P dev

Second, you can use AWS access and secret keys when running TaskCat. Use the -A and -S options to pass the
AWS access key and AWS secret key, respectively, when running TaskCat, as in this example.

$ taskcat -c ci/taskcat.yml -A AXXDEFDFEEFD343D -S ADFSFDFD2343SFDSFSDFSS234SFSD

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 6/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

Exclude files and directories

When you run TaskCat, it syncs the content of your source repository to an S3 bucket (either a temporary bucket
created by TaskCat or a bucket specified by you in the configuration file), and launches the stack for the templates
specified by you from the templates/key prefix within that S3 bucket.

However, in that same source repository, you might have other assets that are not needed by the CloudFormation
template, such as license files, copyright notices, static images, readme files, etc. You can exclude those assets from
uploading to the S3 bucket by using the -e option. You can exclude whole directories or specific files. You can also
use multiple exclude options in a single command. For example, the following command will exclude the images
directory and all text files.

Python
$ taskcat -c ci/taskcat.yml -e images -e *.txt

TROUBLESHOOT FAILED stack DURING testing

When you run TaskCat to test your CloudFormation templates, TaskCat launches the stacks according to your test
configurations and then deletes them, whether the stack creation completed successfully or failed. However, if you
want to keep the stack, for debugging or post deployment
deploy verification, either after it has created successfully or
failed, TaskCat gives you two options.

Option -n retains the stack irrespective of whether the stack creation was successful or failed.

$ taskcat -c ci/taskcat.yml -n

Option -N retains the stack only if the stack creation failed. On successful creation, the stack will be deleted.

$ taskcat -c ci/taskcat.yml -N

Conclusion

In this post, we covered different parameter files, configuration files, and command line options available in TaskCat.
We started with the parameter file and how you can override those parameters using parameter override files. Then I
showed how you can configure your tests in the TaskCat configuration file. Following that, we covered how to
package Lambda functions that you are deploying
deploy as part of your CloudFormation template. Finally, we covered a
few key command line options that you can use to customize the behavior of Taskcat when running your tests.

I hope you enjoyed this post and learned some advance features of TaskCat. As always, let us know your feedback in
the comments below.

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 7/8
5/22/2021 A deep dive into testing with TaskCat | Infrastructure & Automation

TAGS: Automation, AWS CloudFormation, TaskCat, Testing

https://aws.amazon.com/blogs/infrastructure-and-automation/a-deep-dive-into-testing-with-taskcat/ 8/8