Bringing Trust and Transparency in AI

Lab Scenario
Traditional lenders are under pressure to expand their digital portfolio of financial services
to a larger and more diverse audience, which requires a new approach to credit risk
modeling. Their data science teams currently rely on standard modeling techniques - like
decision trees and logistic regression - which work well for moderate datasets, and make
recommendations that can be easily explained. This satisfies regulatory requirements that
credit lending decisions must be transparent and explainable.

To provide credit access to a wider and riskier population, applicant credit histories must
expand beyond traditional credit, like mortgages and car loans, to alternate credit sources
like utility and mobile phone plan payment histories, plus education and job titles. These
new data sources offer promise, but also introduce risk by increasing the likelihood of
unexpected correlations which introduce bias based on an applicant’s age, gender, or
other personal traits.

The data science techniques most suited to these diverse datasets, such as gradient
boosted trees and neural networks, can generate highly accurate risk models, but at a
cost. Such "black box" models generate opaque predictions that must somehow become
transparent, to ensure regulatory approval such as Article 22 of the General Data
Protection Regulation (GDPR), or the federal Fair Credit Reporting Act (FCRA) managed
by the Consumer Financial Protection Bureau.

The credit risk model provided in this tutorial uses a training dataset that contains 20
attributes about each loan applicant. Two of those attributes - age and sex - can be tested
for bias. For this tutorial, the focus will be on bias against sex and age.

We will be using IBM Watson OpenScale for this tutorial. Watson OpenScale allows
enterprises to automate and operationalize AI lifecycle in business applications, ensuring
AI models are free from bias, can be easily explained and understood by business users,
and are auditable in business transactions. Watson OpenScale supports AI models built
and run in the tools and model serve frameworks of your choice.

Watson OpenScale will monitor the deployed model's propensity for a favorable outcome
("No Risk") for one group (the Reference Group) over another (the Monitored Group). In
this tutorial, the Monitored Group for sex is female, while the Monitored Group for age is
19 to 25.
Tutorial objectives
In this tutorial, you will:

• Set up a Watson Studio project, and create, train and deploy a machine learning
model
• Configure and explore trust, transparency and explainability for your model

Provision prerequisite IBM Cloud services

In addition to Watson OpenScale, to complete this tutorial, you need the following
accounts and services.

Important: For best performance, it's recommended that the prerequisite services are
created in the same region as Watson OpenScale. To view available locations for
Watson OpenScale, see Service availability.

1. Log in to your IBM Cloud account with your IBMid.

2. You should see the Dashboard page for IBM Cloud. If not navigate to Dashboard
from the left menu.

3. Click on Services to see list of services already provisioned for this account.
Under services you should see at least 2 services – Watson Studio and Watson
Machine Learning. Watson Studio is a development environment for data
science. You can build your models using different IDEs available in Watson
Studio. Watson Machine Learning is the runtime environment where you can
deploy those models.
 4. Click on the Watson Studio link to navigate to Watson Studio page. Click on Get
started to launch Watson Studio

Set up a Watson Studio project

1. Log in to your Watson Studio account and begin by creating a new project.
Select Create a project.

2. Select Create empty project.

3. Give your project a name and description and click Create.

Associate your IBM Cloud Services with your Watson project

1. Once you have successfully created a Watson Studio project, you will be
redirected to the project overview page. The project is organized in terms of
Assets, Environments, Jobs, Bookmarks, Deployments, Access Control and
Settings.

2. Open your Watson Studio project and select the Settings tab. Scroll down to the
Associated Services section.
3. Click the Add service menu and select Watson.

List of available services are displayed

 4. Click the Add link on the Machine Learning tile and select the Existing tab.
Choose the service you created in the previous section from the Existing Service
Instance menu and click Select.

Add the Credit Risk model

1. Since the primary objective of this exercise is t monitor a deployed model for
accuracy and fairness, we shall skip the step of actually building the model and
will used and existing pre-built model for this purpose. If you so desire, you can
build a model from scratch and deploy it as well.
2. In Watson Studio, select the Assets tab of your project, scroll down to the
Watson Machine Learning Models section, and click the New Watson Machine
Learning model button.
3. From the Select model type section, select From sample and the Credit Risk
model, and then click Create.
Deploy the Credit Risk model

1. From the credit-risk model that you just created from the sample models, Click
the Deployments tab, then select Add Deployment.
2. Enter credit-risk-deploy as the name for your deployment and select the Web
service deployment type.
3. Click Save.
 4. Refresh the page in few minutes to verify the deployment status – should change
from INITIALIZING to DEPLOY_SUCCESS

Configure Watson OpenScale

Provision a Watson OpenScale service

Now that the machine learning model has been deployed, you can configure Watson
OpenScale to ensure trust and transparency with your models.

1. Go to cloud.ibm.com and then navigate to the catalog and select Watson

Openscale service under AI

2. Under Price plans, choose the free Lite plan and click on Create
3. Wait till the service gets provisioned. Once it is provisioned, select the Manage
tab of your Watson OpenScale instance, and click the Launch application button.

4. The IBM® Watson OpenScale Getting Started page opens. Click Auto Setup
 5. Wait till the initialization & setup completes.
6. Follow the Watson Openscale tour to get an appreciation of how Openscale
explains accuracy and fairness

• Learn more about viewing and interpreting the data and monitoring explainability.

Understand fairness and accuracy

1. Overview dashboard showing summary view of all deployed models. Click Next
to proceed

2. View model metrics

3. Monitored group vs reference group

4. Fairness score

5. Fairness over time

6. Detailed view

7. Behaviour based on different data sets

8. Percentage of favourable outcomes

9. Payload data set

10. Performance based on training data

11. Lack of training data

12. Performance based on Debiased dataset
13. Transaction view and explain ability

14. Current vs debiased model

15. Explain a transaction

16. Contributing factors – Advanced explanation

17. The effect of change

18. Feel free to navigate and understand how Openscale has setup the parameters
and shows the result of the analysis