Enquizit Inc.

Integrating Shibboleth and AWS (Runbook)

Version 1.0.0
Nov 2021
Muhammad Saud

1|Page
Table of Contents
AWS Oracle Database Migration Runbook.................................................................................................................................................................................... 1
1. Introduction........................................................................................................................................................................................................................... 3
2. Deployment........................................................................................................................................................................................................................... 3
3. Pre-Migration Configuration.................................................................................................................................................................................................. 4
4. AWS Services & Solution workflow........................................................................................................................................................................................ 7
Step by Step automated Oracle database migration flow:........................................................................................................................................................8
5. Post-Migration Check & Validation...................................................................................................................................................................................... 10
6. Lambdas Layers................................................................................................................................................................................................................... 10
CustomScriptsAsModules........................................................................................................................................................................................................ 10
 eq_oracle..................................................................................................................................................................................................................... 10
 eq_notification............................................................................................................................................................................................................. 10
 eq_file_io..................................................................................................................................................................................................................... 10
 eq_db_config............................................................................................................................................................................................................... 10
CxOracle................................................................................................................................................................................................................................... 10
OracleInstantClient.................................................................................................................................................................................................................. 10
PyYaml..................................................................................................................................................................................................................................... 11
7. StepFunctions workflow...................................................................................................................................................................................................... 11
8. Clean-up............................................................................................................................................................................................................................... 12

2|Page
1. Introduction

This document contains instructions to Integrate Shibboleth Identity provider with AWS migrate. The Runbook assumens that you have already Shibboleth
installed and working in your environment and focuses mainly on configuring Shibboleth IDP with AWS.

This document intended to help anyone who have somewhat knowhow about Shibboleth, SAML, and AWS. The Runbook will helps the desired person to
successfully configure Shibboleth to sign in AWS Console.

2. Deployment

Download code and deployment packages from GitHub and use Cloudformation templates to deploy all the resources. Follow
below steps to start deployment.

1. Deploy VPC and it's related resources like Private & Public subnets, Internet gateway, Nat gateway using 1- Create
VPC.yaml file.
2. Deploy S3 bucket that is required for Lambda Code and Lambda Layer files using 2- Create S3 for Lambda files.yaml file.
Once bucket is created, upload LamdaCode & LambdaLayers directories to this bucket.
3. Deploy Callback-urls application using 3- Serverlessrepo-sfn-callback-urls.yaml  this application will be used to create URLs
to approve or reject export and import.
4. Deploy Lambdas, Secrets and SNS along with its required IAM policy and Roles using 4- Create Lambda.yaml file.
5. Deploy Step Functions and its Workflow along with IAM role and policy using 5- Create StepFunctions.yaml file.

After finishing the deployment navigate to SecretManager console and update secrets/credentials of source and target databases

Update db-migration-params.yaml file as per databases configuration and update SNSTOPICARN which was created at step# 4 part
of Lambda deployment. After updating this file upload, it to oracle-db-migration-meta-data-logs bucket which was also created
at Step# 4.

3|Page
3. Pre-Migration Configuration

Following are the prerequisite configuration steps that needs before starting any migration:

1. Add Source and Target database credentials at Secret Manager.

2. Create “eqdblink” db_link at Source database that connect Source database with Target database for later use. Following SQL statement can be
used to create a db_link.
----Create EQDBLINK
CREATE DATABASE LINK eqdblink
CONNECT TO <target_db_user>
IDENTIFIED BY <target_db_user_password>
USING '(DESCRIPTION= (ADDRESS=(PROTOCOL=TCP)(HOST=<endpoint or IP address of target db>)(PORT=<1521>)
(CONNECT_DATA=(SERVICE_NAME=<target_dbname>)) )';
3. Test db link using following SQL statement.
select sysdate from dual@ eqdblink;
4. If above method will not work due to some reason, then create a base db link and add a DNS entry at target database and then create “ eqdblink”
using base db link SQL statements are as following:
----Create BASE_DBLINK
CREATE DATABASE LINK BASE_DBLINK
CONNECT TO <target_db_user>
IDENTIFIED BY <target_db_user_password>
USING '(DESCRIPTION= (ADDRESS=(PROTOCOL=TCP)(HOST=<endpoint or IP address of target db>)(PORT=<1521>)
(CONNECT_DATA=(SERVICE_NAME=<target_dbname>)) )';
----Add TNS entry at TNSNAMES.ORA file at server like below
base_dblink =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = <endpoint or IP address of target db>)(PORT = <1521>))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORCL)
)
)
4|Page
 ---Now create eqdblink using BASE_DBLINK
CREATE DATABASE LINK eqdblink
CONNECT TO <target_db_user> IDENTIFIED BY <target_db_user_password>
USING 'BASE_DBLINK';
5. Drop any test dblink using following SQL statement:
--Drop dblink
DROP DATABASE LINK testdblink;
6. “eqdblink” is hardcoded at db-migration-params.yml configuration file so, do not change the db_link name. If db_link name changed, then dump
will not transfer to Target db.
7. Add Schemas and LOB tables that needs to be exported at db-migration-params.yml configuration file, below is the example snapshot for
reference.

5|Page
 8. Update datapump export configuration at db-migration-params.yml configuration file, below is the example snapshot for reference.

9. Update datapump import configuration at db-migration-params.yml configuration file, below is the example snapshot for reference.

6|Page
 10. Update SNSTopicARN at the end of the db-migration-params.yml configuration file, below is the example snapshot for reference.

4. AWS Services & Solution workflow

 AWS Lambda & Lambda Layers

 Secret Manager
 S3
 VPC
 SNS
 API Gateway
 Step Functions
 CloudFormation

7|Page
 Step by Step automated Oracle database migration flow:

1. Make sure Source database is connected to AWS VPC and network configuration is up to date.
2. Update db-migration-params.yml configuration file and upload it to s3 bucket.
3. Start execution of AWS Step function which will invoke AWS Lambda functions associated to it.
4. Step functions invoke first AWS Lambda eq_db_testcxn to check/test both source and target database connections using credentials provided at
Secret manager. If there are missing or wrong secrets either source or target database, then connection will fail and AWS Step function workflow
end execution.
5. If test connection works then step function invoke eq_db_source_properties Lambda and get Meta_Data in the form of SQL statements from
source database including users, roles, profiles, tablespaces, directories, tablespace-grants, synonyms, role-grants, object-grants, system-grants and
save these SQL files at S3 bucket.
6. Files including Meta_Data (SQL statements files) initialize & finalized execution Meta_Data SQL statements logs, validation csv files will be stored on
S3 bucket using dynamic path <s3-bucket-name>/datafiles/<source-db-name>/

8|Page
 7. AWS Step functions invoke eq_db_export Lambda function and start exporting database using data_pump utility, .dmp files will be stored at Source
database data_pump directory. Data_pump export configuration will be fetched from db-migration.param.yml configuration file.
8. Once export finished, AWS Step functions invoke SendApprovalEmail Lambda function, it will trigger another Lambda function(sfn-callback-urls
app) in backed to create callback-url for step functions using APIGateway.
9. The sfn-callback-urls app generate one-time-use callback URLs through a call to Amazon API Gateway.
10. Email will be sent to SNS subscriber (migration engineer) for approving or rejecting the Step functions workflow, this will help migration engineer to
review export logs and if there are no errors and export has finished without any error then click on approve URL in the email otherwise choose
reject URL to cancel migration and resolve export issues to continue.
11. Once user (migration engineer) approves the export then next function eq_db_target_initialize will be invoked to initialized target database before
starting import. It will execute SQL statements to create users, roles, profiles, tablespaces, and directories at target database.
12. While executing initialized SQL statements if there are any errors then these errors and a summary of these errors will be logged in a file and once
finished the SQL execution these log files will be uploaded to S3 bucket for resolving those errors later-on.
13. eq_db_transfer_files will be invoked to transfer .dmp files directly from Source database to Target database using “eqdblink” db_link which should
be created as part of pre-migration step at Source database.
14. After transferring dump files to data_pump directory at Target database, AWS step function will invoke eq_db_import Lambda to start import at
Target database. All the import related data_pump configuration will be fetched from db-migration-params.yml configuration file.
15. Once import finished, AWS Step functions again invoke SendApprovalEmail Lambda function to user for approving or rejecting the migration flow
at this stage, it will trigger another Lambda function(sfn-callback-urls app) in backed to create callback-url for step functions using APIGateway.
16. The sfn-callback-urls app generate one-time-use callback URLs through a call to Amazon API Gateway.
17. Email will be sent to SNS subscriber (migration engineer) for approving or rejecting the Step functions workflow, this will help migration engineer to
review import logs and if there are no errors and import has finished without any error then click on approve URL in the email otherwise choose
reject URL to cancel migration and resolve import issues to continue.
18. At this stage export and import finished smoothly, eq_db_target_finalize Lambda will be invoked to finalized target database after import. It will
execute SQL statements of tablespace-grants, created synonyms, role-grants, object-grants, and system-grants at target database.
19. While executing finalized SQL statements if there are any errors then these errors and a summary of these errors will be logged in a file and once
finished the SQL execution these log files will be uploaded to S3 bucket for resolving those errors later-on.
20. This is the last Lambda function which will be invoked if all the above executions finished without any major errors, eq_validation will be invoked to
validate the migration. This function validates Source and Target object-count, Lob-Size, Privileges-count, Table-count, Synonym-count, and User-
count and log all these number into a .csv file.
21. Validation files will be uploaded to S3 bucket at same datafiles path.

5. Post-Migration Check & Validation

9|Page
Once migration finished then it’s time to validate and check if there are any minor errors that needs to be fixed. Sometime due to insufficient privileges
some SQL statements related to Meta_Data would not execute during the migration like synonyms, roles, profiles, users, and views. So, it’s important to go
through validation files and see if Source database count is not matching with Target database, then review initialized & finalized logs.

Contact Client database team to assign sufficient privileges to source user and extract Meta_data manually against specific errors and execute it at Target
database.

6. Lambdas Layers
There are four Lambda Layers that are attached with all the Lambdas excluding SendApprovalEmail function. Following are the list of Layers and their
modules.

CustomScriptsAsModules
 eq_oracle
 Include methods to perform SQL executions for several datapump functionalities like start datapump export/import, check datapump
status, check file transfer status etc.

 eq_notification
 Include methods to use AWS SNS service for sending notification during the migration.

 eq_file_io
 Include methods to handle file input and output like write/read data from files such as SQL statement, logs etc.

 eq_db_config
 Include configuration methods that linked with db-migration-params.yml configuration file.

CxOracle
 It is a Python extension module that enables access to Oracle Database. 

OracleInstantClient
 Oracle Instant Client enables development and deployment of applications that connect to Oracle Database, either on-premises or in
the Cloud.

PyYaml
 It is a YAML parser and emitter for Python.

10 | P a g e
7. StepFunctions workflow

8. Clean-up

11 | P a g e
Once Oracle databases migrated to AWS then it’s time to clean-up all the resources. Visit AWS CloudFormation console and delete all the stacks that
deployed at deployment stage.

Remember, to delete data from S3 buckets before starting deletion of CloudFormation stacks.

12 | P a g e