Concourse is a pipeline-based continuous thing-doer.

The word "pipeline" is all the rage in CI these days, so being more specific about this
term is kind of important; Concourse's pipelines are significantly different from the
rest.

Pipelines are built around Resources, which represent all external state, and Jobs,
which interact with them. Concourse pipelines represent a dependency flow, kind of
like distributed Makefiles. Pipelines are designed to be self-contained so as to
minimize server-wide configuration. Maximizing portability also mitigates risk, making
it easier for projects to recover from CI disasters.

Resources like the git resource and s3 resource are used to express source code,
dependencies, deployments, and any other external state. This interface is also used
to model more abstract things like scheduled or interval triggers, via
the time resource.

Resource Types are defined as part of the pipeline itself, making the pipelines more
self-contained and keeping Concourse itself small and generic without resorting to a
complicated plugin system.

Jobs are sequences get, put, and task steps to execute. These steps determine the
job's inputs and outputs. Jobs are designed to be idempotent and loosely coupled,
allowing the pipeline to grow with the project's needs without requiring engineers to
keep too much in their head at a time.

Everything in Concourse runs in a container. Instead of modifying workers to install

build tools, Tasks describe their own container image (typically using Docker images
via the registry-image resource).

1.4 Pipelines
A pipeline is the result of configuring Jobs and Resources together. When you
configure a pipeline, it takes on a life of its own, to continuously detect resource
versions and automatically queue new builds for jobs as they have new available
inputs.

Pipelines are configured as declarative YAML files, fitting the following schema:

jobs: [job]
A set of jobs for the pipeline to continuously check.
resources: [resource]
A set of resources for the pipeline to continuously check.
resource_types: [resource_type]
A set of resource types for resources within the pipeline to use.
groups: [group]
A list of groups to use for cleaning up/organizing jobs in the web UI.

Each group must have a unique name.

1.4.1 Setting Pipelines

Pipelines are configured entirely via the fly CLI. There is no GUI.

fly set-pipeline
To submit a pipeline configuration to Concourse from a file on your local disk you
can use the -c or --config flag, like so:

$ fly -t example set-pipeline \ --pipeline my-pipeline \ --config pipeline.yml

This will present a diff of the changes and ask you to confirm the changes. If you
accept then Concourse's pipeline configuration will switch to the pipeline definition in
the YAML file specified.

Pipeline ((vars))
The pipeline configuration can contain template variables in the form of ((foo-bar)).
They will be replaced with values populated by repeated --var, --yaml-var, or --load-
vars-from flags, or at runtime via a credential manager.

This allows for credentials to be extracted from a pipeline config, making it safe to
check in to a public repository or pass around.

For example, if you have a pipeline.yml as follows:

resources: - name: private-repo type: git source: uri: git@... branch: master
private_key: ((private-repo-key))
...you could then configure this pipeline like so:

$ fly -t example set-pipeline \ --pipeline my-pipeline \ --config pipeline.yml \ --var

"private-repo-key=$(cat id_rsa)"
Or, if you had a credentials.yml as follows:

private-repo-key: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY---
--
...you could configure it like so:

$ fly -t example set-pipeline \ --pipeline my-pipeline \ --config pipeline.yml \ --load-

vars-from credentials.yml
Concatenation is supported for string values - foo-((var)) with var: bar will resolve
to foo-bar.
If both --var/--yaml-var and --load-vars-from are specified, the --var and --yaml-var flags
take precedence.

Values other than strings (e.g. bools, arrays) may also be specified by using either --
yaml-var or --load-vars-from.

fly validate-pipeline
To validate a local pipeline configuration without submitting it to Concourse,
run validate-pipeline:

$ fly validate-pipeline --config pipeline.yml

By default, pipeline errors will cause validate-pipeline to fail, but warnings won't. To
fail on both errors and warnings, pass the `--strict` flag.

fly format-pipeline
To format a pipeline config in a "canonical" form (i.e. keys are in normal order,
with name first for example), run:

$ fly format-pipeline --config pipeline.yml

This will print the formatted pipeline config to stdout. To update the file in-place,
pass --write/-w.