Services Provisioning Python Application

Background:

This Python application will be the main provisioning system for our product. It needs to support being
run “Locally” on ones workstation. The following high level requirements are needed from the Python
Application.

High Level Workflow:

1. Provision GitLab resources (Can be done in parallel)

2. Provision CloudFlare resources (Can be done in parallel)
3. Link GitLab repo resources with CloudFlare pages resources (steps one and two should be confirmed
completed before this step)
4. Clone the template files from the defined template GitLab repo’s locally and clone into newly
provisioned repositories.
5. Test that the CloudFlare pages content is live with an HTTP endpoint request on the linked content
from the GitLab repo.
6. Send a formed URL REST response back to another endpoint in CloudFlare once provisioning is
complete.

Python Requirements:

1. Preferred you use the attached library (GitHelper.py) for any ‘git‘ needs locally or in the pipeline.
2. All environment variables must be set using the “configargparser” library for python.
3. Logging should be enabled throughout the code (see attached example).
4. Classes and methods should be used as much as possible in order to minimize code.
5. Everything should be recorded for how long a process takes in the Python application with the
“time” module.
6. Anything that can be done in parallel should be done so and will be noted below.
7. Pylint tests should be written to test each step or function in the code.

Python ConfigArgParser Environment Variables Needed:

COMPANY_URL (Required)
This will be a unique domain name of a company website (i.e. www-whatever-com)

COMPANY_ID (Required)
This will be a unique UUID passed into the provisioner to identify content in the repo.

PROVISION_TYPE (Required)
This will dictate how we name the group in Gitlab as well as the name of the CloudFlare pages
either with a suffix or without a suffix.

LEED_API_KEY
A unique API key use for our internal product analytics needs that will be inserted/updated into
a file content in a provisioned repo from a template repo.
GIT_API_TOKEN
The GitLab API security token used to provision resources on Gitlab

CF_API_TOKEN
The CloudFlare API security token used to provision resources on CloudFlare

TEMPLATE_REPO (Default to use TBD)

A repo name or ID that will be used to source a set of template files to be copied into newly
created repositories under groups created for a customer by the COMPANY_URL. There could be
multiple template repo’s defined at a later date (i.e. TEMPLATE_REPO1, TEMPLATE_REPO2) or
possibly just a top level repo that you clone from that you will copy different files from directories into
different repo’s.

PROVISION_SUFFIX
A suffix that will be appended only when the PROVISION_TYPE has a value of “TRIAL”. It
should have a default of testdrive.company.com. It should be able to be overridden if another is
specified and the default easily changed in the code if wanted/needed later.

The application needs and ENV called company-url, company-id and provision-type (written for
configargparser, COMPANY_URL and COMPANY_ID and PROVISION_TYPE will be the ENV
variables to look for if nothing is passed). It is a required field and must fail if one or any is not
specified. The other variables defined above will be used throughout the application provision process
either for authentication or to have configurable ENV’s in the future.

Provision GitLab Resources:

1. An application in CloudeFlare will send a Web-hook call to a project in Gitlab (For now we will
assume it is a project called “Automation Tools” within a the top-level Gitlab of Company and Group
of “Operations (i.e. company-ai/operations/automation-tools). THIS IS OUT OF SCOPE FOR
YOUR WORK. With the following environment variables sent in the REST API Web-hook payload of
COMPANY_ID, COMPANY_URL and PROVISION_TYPE.

2. Use Gitlab REST API’s to first check if the Subgroup already exists, if so error out with it already
existing and if not create a Groups/Subgroups and repositories after this step. The main reason for this
is to be able to do local testing. This will require a global configargparser of –git-api-token to be set for
authentication see:
https://docs.gitlab.com/ee/api/api_resources.html
3a. A current group of Company/Customers (i.e. company-ai/customers) is and will always be
pre-created.
3b. A check should be performed on the PROVISIONG_TYPE that was passed in, if it is either
TRIAL or PROD
- If the provisioned type is PROD then create a new subgroup with the COMPANY_URL under
the Company/Customers group. (i.e. comapny/customers/www-whatever-com)
- If the provisioned type is TRIAL then create a new subgroup of COMPANY_URL-
PROVISION_SUFFIX (note the suffix should be defined by either a global ENV or a configargparse
with a value of “testdrive-company.com” as the default. But have the ability to change the default in
the code as well change it with a new configargparse passed in at runtime (i.e.
Company/Customers/www-whatever-com-testdrive-company.com).
*Note: We have a future requirement to just create it based off of the domain name (i.e. whatever in
the www-whatever-com). So this may mean we strip out whatever is after the www. And before .com
or www- and com in the example provided. So please make a function or method to clean up the
COMPANY_URL to just top level domain (i.e. the “whatever” in www-whatever-com)

Create a subgroup in Gitlab

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \

--header "Content-Type: application/json" \
--data '{"path": "<subgroup_path>", "name": "<subgroup_name>", "parent_id": <parent_group_id>
}' \
"https://gitlab.example.com/api/v4/groups/"

3c. Once the Group is created several repositories need to created under the above group of:
“Analytics Content” (analytics-content)
“Preview Site” (preview-site)
“Public Site” (public-site)
“Raw Content” (raw-content)
“Staging Content” (staging-content)

Create a project in Gitlab

curl --request POST --header "PRIVATE-TOKEN: <your-token>" \

--header "Content-Type: application/json" --data '{
"name": "new_project", "description": "New Project", "path": "new_project",
"namespace_id": "42", "initialize_with_readme": "true"}' \
--url 'https://gitlab.example.com/api/v4/projects/'

3d. Once repo’s are created a clone from another set of projects will clone a template into
certain repo’s. There needs to be a check that that template repo(s) are there first and then run the step
to clone the files into the new repo. Recommend you put a place holder in the code for when we have
the hierarchical layout for the source template repos to be cloned into these destination repo’s. For
example we plan on cloning content from something like:
Company/Operations/Templates/Customers/Staging Content → Company/Operations/www-
whatever-com/staging-content
Company/Operations/Templates/Customers/Preview Content→ Company/Operations/www-
whatever-com/preview-content

We are also planning on using the attached GitHelper.py code to do the cloning of the repo’s stored as a
library in the directory structure of the project (i.e. lib/GitHelper.py) Note the .py is an extension to
designate that it is a Python program.

3e. Update the filed called “identify.json” in the “Raw Contents” repo in the
Company/Customers/www-whatever-com (this is just an example) with the following:
Below are the environment variables specified with configargparser.
{
"companyId": "COMPANY_ID",
"leedAPIKey": "LEED_APY_KEY"
}

3f. We will have other files that need to be updated. Please create a
method that can do this for different definitions of repo’s and files/folders that
need to be updated.

Again this is planned to be pulled from a template set of files from some template repo’s but the file
will need to be updated and there will be other files they will need to be updated so I recommend
creating a whole section of updating files to be able to update multiple files across multiple repo’s we
just created.

CloudFlare Provisioning Process:

1. First check if the 2 CloudFlare Pages are already there, if note create 2 CloudFlare Pages based on
the PROVISION_TYPE and COMPANY_URL see:
https://api.cloudflare.com/#getting-started-requests.
One will have the “preview-” prefix along with the COMPANY-URL or with the SUFFIX (i.e.
preview-www-whatever-com or preview-www-whatever-com-testdrive-domain-com. The other
CloudFlare page will have just the COMPANY_URL or COMPANY_URL-PROVISION_SUFFIX (i.e.
www-whaterver-com or www-whatever-com-testdrive-company-com).

- If PROVISION_TYPE “TRIAL” was passed in then create a page called COMPANY_URL-

PROVISION_SUFFIX
- If PROVISION_TYPE “PROD” was passed in then create a page called COMPANY_URL

2. Link the CloudFlare page for COMPANY_URL-PROVISION_PREFIX OR COMPANY_URL to

the following repo (again below are just examples):

CFPages/preview-www-whatever-com (or with the suffix) →

GitLab/Company/Customers/COMPANY_URL(or the PROVISION_SUFFIX)/Preview Site.

CFPages/www-whatever-com (or with the suffix)→

GitLab/Company/Customers/COMPANY_URL(or the PROVISION_SUFFIX)/Public Site.

The above CloudFlare Pages should be linked the newly respective named repo’s.

See example below:

CloudFlare Create Pages Project

https://api.cloudflare.com/#pages-project-create-project

POST accounts/:account_identifier/pages/projects
curl -X POST "https://api.cloudflare.com/client/v4/accounts/9a7806061c88ada191ed06f989cc3dac/
pages/projects" \
-H "X-Auth-Email: user@example.com" \
-H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
-H "Content-Type: application/json" \
--data '{"name":"NextJS Blog","id":"7b162ea7-7367-4d67-bcde-1160995d5","created_on":"2017-01-
01T00:00:00Z","subdomain":"helloworld.pages.dev","domains":
["customdomain.com","customdomain.org"],"source":{"type":"github","config":
{"owner":"cloudflare","repo_name":"ninjakittens","production_branch":"main","pr_comments_enabl
ed":true,"production_deployments_enabled":true,"preview_deployment_setting":"custom","preview_b
ranch_includes":["release/*","production","main"],"preview_branch_excludes":["dependabot/
*","dev","*/ignore"]}},"build_config":{"build_command":"npm run
build","destination_dir":"build","root_dir":"/","web_analytics_tag":"cee1c73f6e4743d0b5e6bb1a0bc
aabcc","web_analytics_token":"021e1057c18547eca7b79f2516f06o7x"},"deployment_configs":
{"preview":{"env_vars":{"BUILD_VERSION":{"value":"3.3"}},"kv_namespaces":{"KV_BINDING":
{"namespace_id":"5eb63bbbe01eeed093cb22bb8f5acdc3"}},"durable_object_namespaces":
{"DO_BINDING":{"namespace_id":"5eb63bbbe01eeed093cb22bb8f5acdc3"}},"r2_buckets":
{"R2_BINDING":{"name":"some-bucket"}},"d1_databases":{"D1_BINDING":{"id":"445e2955-951a-
43f8-a35b-a4d0c8138f63"}},"compatibility_date":"2022-01-01","compatibility_flags":
["url_standard"]},"production":{"env_vars":{"BUILD_VERSION":
{"value":"3.3"}},"kv_namespaces":{"KV_BINDING":
{"namespace_id":"5eb63bbbe01eeed093cb22bb8f5acdc3"}},"durable_object_namespaces":
{"DO_BINDING":{"namespace_id":"5eb63bbbe01eeed093cb22bb8f5acdc3"}},"r2_buckets":
{"R2_BINDING":{"name":"some-bucket"}},"d1_databases":{"D1_BINDING":{"id":"445e2955-951a-
43f8-a35b-a4d0c8138f63"}},"compatibility_date":"2022-01-01","compatibility_flags":
["url_standard"]}},"latest_deployment":{},"canonical_deployment":{},"production_branch":"main"}'

Example payload to link CloudFlare page to a Gilab repo:

"source": {
"type": "gitlab",
"config": {
"owner": "company-ai",
"repo_name": "customers/www-whatever-com/public-site",
"production_branch": "main",
"pr_comments_enabled": false
}

2. Test CloudFlare Deployment by running an already created Python application that will be in the
root of the “Automation Tools” (automation-tools) Gitlab Project called ‘siteBuilderPipeline’

For building the “Preview Site” for the CloudFlare page marked as preview. (i.e.
preview.www.whatever.com or with the SUFFIX)
./setupSiteBuilder
./siteBuilderPipeline -v --preview -c ‘COMPANY_URL or COMPANY_URL-PROVISION_SUFFIX’
"Bob the Automated Site Builder Scaffolding"

For building the “Public Site” for the CloudFlare page marked as public site. (i.e. www.whatever.com
or with the SUFFIX)
./setupSiteBuilder
./siteBuilderPipeline -v -c COMPANY_URL or COMPANY_URL-PROVISION_SUFFIX"Bob the
Automated Site Builder Scaffolding"
Test CloudFlare Page is Live

3. Test an endpoint REST call to the preview CloudFlare page and COMPANY_URL page (or with the
suffix). We currently do not have the template content built to test but will soon. i.e. But we should
expect a 200 or some other code to be a success before moving on to the last step.
Send Provision Completion REST Command Back to CloudFlare Endpoint

4. If all succeeds for all steps then we will push a final REST call to a separate CloudFlare page that
our build is complete. Again this is not built yet but it should be something simple that sends the
COMPANY_URL and COMPANY_ID and maybe the PROVISION_TYPE back that it is complete.
Possibly even a status code.