BUDT 725: Models and Applications in Operations Research

by

Bruce L. Golden
R.H. Smith School of Business

Volume 1- Customer Relationship Management Through Data Mining
1

Customer Relationship Management Through Data Mining
Introduction to Customer Relationship Management (CRM)
Introduction to Data Mining

Data Mining Software
Churn Modeling Acquisition and Cross Sell Modeling
2

Relationship Marketing
Relationship Marketing is a Process
communicating with your customers
listening to their responses

Companies take actions
marketing campaigns new products

new channels
new packaging
3

Relationship Marketing -- continued
Customers and prospects respond
most common response is no response

This results in a cycle
data is generated opportunities to learn from the data and improve the

process emerge

4

The Move Towards Relationship Management
E-commerce companies want to customize the user experience
Supermarkets want to be infomediaries Credit card companies want to recommend good restaurants and hotels in new cities Phone companies want to know your friends and family

Bottom line: Companies want to be in the business of serving customers rather than merely selling products
5

CRM is Revolutionary
Grocery stores have been in the business of stocking shelves Banks have been in the business of managing the spread between money borrowed and money lent Insurance companies have been in the business of managing loss ratios Telecoms have been in the business of completing telephone calls

Key point: More companies are beginning to view customers as their primary asset
6

exploiting existing customers becomes more important total customers new customers churners 9 10 11 7 Year .Why Now ? Representative Growth in a Maturing Market Number of Customers (1000s) 1200 1000 800 600 400 200 0 0 1 2 3 4 5 6 7 8 In this region of rapid growth. building infrastructure is more important than CRM As growth flattens out.

route through switching system. shipping method requested.The Electronic Trail A customer places a catalog order over the telephone At the local telephone company time of call. credit card used. long distance company used. call center. … At the catalog items ordered. inventory update. … 8 . promotion response. number dialed. … At the long distance company (for the toll-free number) duration of call.

approval code.continued At the credit card clearing house transaction date.The Electronic Trail-. … Bottom line: Companies do keep track of data 9 . amount charged. available credit update. time stamp at sorting center. vendor number. interest rate. time stamp at truck. … At the bank billing record. … At the package carrier zip code.

its volume fell FedEx identified those customers whose FedEx volumes had increased and then decreased These customers were using UPS again FedEx made special offers to these customers to get all of their business 10 . UPS went on strike FedEx saw its volume increase After the strike.An Illustration A few years ago.

The Corporate Memory Several years ago. you will miss them Victoria’s Secret builds customer loyalty with a no-hassle returns policy some “loyal customers” return several expensive outfits each month they are really “loyal renters” 11 . Land’s End could not recognize regular Christmas shoppers some people generally don’t shop from catalogs but spend hundreds of dollars every Christmas if you only store 6 months of history.

CRM Requires Learning and More Form a learning relationship with your customers Notice their needs On-line Transaction Processing Systems Remember their preferences Decision Support Data Warehouse Learn how to serve them better Data Mining Act to make customers more profitable 12 .

The Importance of Channels Channels are the way a company interfaces with its customers Examples Direct mail Email Banner ads Telemarketing Billing inserts Customer service centers Messages on receipts Key data about customers come from channels 13 .

Channels -.continued Channels are the source of data Channels are the interface to customers Channels enable a company to get a particular message to a particular customer Channel management is a challenge in organizations CRM is about serving customers through all channels 14 .

Where Does Data Mining Fit In? Hindsight Analysis and Reporting (OLAP) Foresight Statistical Modeling Insight Data Mining 15 .

and customer support operations Source: Berry and Linoff (1997) 16 .Our Definition of Data Mining Exploration and analysis of large quantities of data By automatic or semi-automatic means To discover meaningful patterns and rules These patterns allow a company to better understand its customers improve its marketing. sales.

Data Mining for Insight Classification Prediction Estimation Automatic Cluster Detection Affinity Grouping Description 17 .

Finding Prospects A cellular phone company wanted to introduce a new service They wanted to know which customers were the most likely prospects Data mining identified “sphere of influence” as a key indicator of likely prospects Sphere of influence is the number of different telephone numbers that someone calls 18 .

Government to save millions of dollars by not paying fraudulent medical insurance claims 19 .Paying Claims A major manufacturer of diesel engines must also service engines under warranty Warranty claims come in from all around the world Data mining is used to determine rules for routing claims some are automatically approved others require further research Result: The manufacturer saves millions of dollars Data mining also enables insurance companies and the Fed.

Cross Selling Cross selling is another major application of data mining What is the best additional or best next offer (BNO) to make to each customer? E. a bank wants to be able to sell you automobile insurance when you get a car loan The bank may decide to acquire a full-service insurance agency 20 ..g.

more effective retention campaign 21 .Holding on to Good Customers Berry and Linoff used data mining to help a major cellular company figure out who is at risk for attrition And why are they at risk They built predictive models to generate call lists for telemarketing The result was a better focused.

Weeding out Bad Customers Default and personal bankruptcy cost lenders millions of dollars Figuring out who are your worst customers can be just as important as figuring out who are your best customers many businesses lose money on most of their customers 22 .

They Sometimes get Their Man The FBI handles numerous. complex cases such as the Unabomber case Leads come in from all over the country The FBI and other law enforcement agencies sift through thousands of reports from field agents looking for some connection Data mining plays a key role in FBI forensics 23 .

customers are placed into groups Customers in each group are assumed to have an affinity for the same types of products New product recommendations can be generated automatically based on new purchases made by the group This is sometimes called collaborative filtering 24 .Anticipating Customer Needs Clustering is an undirected data mining technique that finds groups of similar items Based on previous purchase patterns.

CRM Focuses on the Customer The enterprise has a unified view of each customer across all business units and across all channels This is a major systems integration task The customer has a unified view of the enterprise for all products and regardless of channel This requires harmonizing all the channels 25 .

and McDonalds CRM tends to focus on the smaller customer -the consumer But.g.A Continuum of Customer Relationships Large accounts have sales managers and account teams E.. Coca-Cola. Disney. small businesses are also good candidates for CRM 26 .

therefore.What is a Customer A transaction? An account? An individual? A household? The customer as a transaction purchases made with cash are anonymous most Web surfing is anonymous we. know little about the consumer 27 .

… Telecommunications long distance. … Insurance auto policy. homeowners. local. … Utilities The account-level view of a customer also misses the boat since each customer can have multiple accounts 28 . a customer is an account Retail banking checking account. life insurance. mortgage.A Customer is an Account More often. mobile. ISP. auto loan.

Customers Play Different Roles Parents buy back-to-school clothes for teenage children children decide what to purchase parents pay for the clothes parents “own” the transaction Parents give college-age children cellular phones or credit cards parents may make the purchase decision children use the product It is not always easy to identify the customer 29 .

… Retirement sell home. divorce. travel. hobbies. … Family Life marriage. buy house. school. move away from parents. … Much marketing effort is directed at each stage of life 30 .The Customer’s Lifecycle Childhood birth. … Young Adulthood choose career. graduation. children.

The Customer’s Lifecycle is Unpredictable
It is difficult to identify the appropriate events
graduation, retirement may be easy marriage, parenthood are not so easy many events are “one-time”

Companies miss or lose track of valuable information
a man moves a woman gets married, changes her last name, and merges her accounts with spouse

It is hard to track your customers so closely, but, to the extent that you can, many marketing opportunities arise
31

Customers Evolve Over Time
Customers begin as prospects Prospects indicate interest
fill out credit card applications apply for insurance visit your website

They become new customers
After repeated purchases or usage, they become established customers Eventually, they become former customers
either voluntarily or involuntarily
32

Business Processes Organize Around the Customer Lifecycle
Acquisition Activation Relationship Management

Winback Former Customer

High Value Prospect New Customer Established Customer Voluntary Churn

High Potential

Low Value

Forced Churn
33

Different Events Occur Throughout the Lifecycle
Prospects receive marketing messages When they respond, they become new customers They make initial purchases They become established customers and are targeted by cross-sell and up-sell campaigns Some customers are forced to leave (cancel) Some leave (cancel) voluntarily Others simply stop using the product (e.g., credit card) 34 Winback/collection campaigns

Different Data is Available Throughout the Lifecycle
The purpose of data warehousing is to keep this data around for decision-support purposes
Charles Schwab wants to handle all of their customers’ investment dollars Schwab observed that customers started with small investments

35

Different Data is Available Throughout the Lifecycle -- continued
By reviewing the history of many customers, Schwab discovered that customers who transferred large amounts into their Schwab accounts did so soon after joining After a few months, the marketing cost could not be justified

Schwab’s marketing strategy changed as a result
36

Different Models are Appropriate at Different Stages Prospect acquisition Prospect product propensity Best next offer Forced churn Voluntary churn Bottom line: We use data mining to predict certain events during the customer lifecycle 37 .

turn-key software solutions packages have generic churn models & response models they work pretty well Master Data Mining develop expertise in-house use sophisticated software such as Clementine or Enterprise Miner 38 .Different Approaches to Data Mining Outsourcing let an outside expert do the work have him/her report the results Off-the-shelf.

Privacy is a Serious Matter Data mining and CRM raise some privacy concerns These concerns relate to the collection of data. more than the analysis of data The next few slides illustrate marketing mistakes that can result from the abundance and availability of data 39 .

who had diabetes to get eye exams the IT group was asked for a list of members with diabetes 40 .Using Data Mining to Help Diabetics Early detection of diabetes can save money by preventing more serious complications Early detection of complications can prevent worsening retinal eye exams every 6 or 12 months can prevent blindness these eye exams are relatively inexpensive So one HMO took action they decided to encourage their members.

thinking the diabetes had recurred She threatened to sue the HMO Mistake: Disconnect between the domain expertise and data expertise 41 .One Woman’s Response Letters were sent out to HMO members Three types of diabetes – congenital. gestational One woman contacted had gestational diabetes several years earlier She was “traumatized” by the letter. adult-onset.

Gays in the Military The “don’t ask. don’t tell” policy allows discrimination against openly gay men and lesbians in the military Identification as gay or lesbian is sufficient grounds for discharge This policy is enforced Approximately 1000 involuntary discharges each year 42 .

he listed “gay” A colleague discovered the account and called AOL to verify that the owner was McVeigh AOL gave out the information over the phone McVeigh was discharged (three years short of his pension) The story doesn’t end here 43 . McVeigh used an AOL account. with an anonymous alias Under marital status.The Story of Former Senior Chief Petty Officer Timothy McVeigh Several years ago.

Two Serious Privacy Violations AOL breached its own policy by giving out confidential user information AOL paid an undisclosed sum to settle with McVeigh and suffered bad press as well The law requires that government agents identify themselves to get online subscription information This was not done McVeigh received an honorable discharge with full retirement pension 44 .

Friends. then calls to them would be discounted Did MCI have to ask customers about who they call regularly? Early in 1999. Family. MCI promoted the “Friends and Family” program They asked existing customers for names of people they talked with often If these friends and family signed up with MCI. and Others In the 1990s. BT (formerly British Telecom) took the idea one step beyond BT invented a new marketing program discounts to the most frequently called numbers 45 .

BT Marketing Program BT notified prospective customers of this program by sending them their most frequently called numbers One woman received the letter uncovered her husband’s cheating threw him out of the house sued for divorce The husband threatened to sue BT for violating his privacy BT suffered negative publicity 46 .

legal.No Substitute for Human Intelligence Data mining is a tool to achieve goals The goal is better service to customers Only people know what to predict Only people can make sense of rules Only people can make sense of visualizations Only people know what is reasonable. tasteful Human decision makers are critical to the data mining process 47 .

A Long. Long Time Ago There was no marketing There were few manufactured goods Distribution systems were slow and uncertain There was no credit Most people made what they needed at home There were no cell phones There was no data mining It was sufficient to build a quality product and get it to market 48 .

.g. there has been an explosion in the number of products in the last 50 years Now. we need to anticipate and create demand (e. e-commerce) This is what marketing is all about 49 . a typical grocery store carried 800 different items A typical grocery store today carries tens of thousands of different items There is intense competition for shelf space and premium shelf space In general.Then and Now Before supermarkets.

Effective Marketing Presupposes High quality goods and services Effective distribution of goods and services Adequate customer service Marketing promises are kept Competition direct (same product) “wallet-share” Ability to interact directly with customers 50 .

The ACME Corporation Imagine a fictitious corporation that builds widgets It can sell directly to customers via a catalog or the Web maintain control over brand and image It can sell directly through retail channels get help with marketing and advertising It can sell through resellers outsource marketing and advertising entirely Let’s assume ACME takes the direct marketing approach 51 .

Chinese porcelain. Bruges cloth really took off in the 20th Century Advertising is hard media mix problem – print. TV.” 52 .Before Focusing on One-to-One Marketing Branding is very important provides a mark of quality to consumers old concept – Bordeaux wines. radio. billboard. Web difficult to measure effectiveness “Half of my advertising budget is wasted. I just don’t know which half.

000 responses/week) 53 .Different Approaches to Direct Marketing Naïve Approach get a list of potential customers send out a large number of messages and repeat Wave Approach send out a large number of messages and test Staged Approach send a series of messages over time Controlled Approach send out messages over time to control response (e.g.. get 10.

The World is Speeding Up Advertising campaigns take months market research design and print material Catalogs are planned seasons in advance Direct mail campaigns also take months Telemarketing campaigns take weeks Web campaigns take days modification/refocusing is easy 54 .

How Data Mining Helps in Marketing Campaigns Improves profit by limiting campaign to most likely responders Reduces costs by excluding individuals least likely to respond AARP mails an invitation to those who turn 50 they excluded the bottom 10% of their list response rate did not suffer 55 .

etc. with inventory control.How Data Mining Helps in Marketing Campaigns--continued Predicts response rates to help staff call centers. Identifies most important channel for each customer Discovers patterns in customer data 56 .

000 Best estimates indicate between 1 and 10 million customers ACME wants to target the customer base costeffectively ACME seeks to assign a “score” to each customer which reflects the relative likelihood of that customer purchasing the product 57 .Some Background on ACME They are going to pursue a direct marketing approach Direct mail marketing budget is $300.

How Do You Assign Scores Randomly – everyone gets the same score Assign relative scores based on ad-hoc business knowledge Assign a score to each cell in an RFM (recency. potential customers Build a predictive model based on similar product sales in the past 58 . monetary) analysis Profile existing customers and use these profiles to assign scores to similar. frequency.

Some scores may be the same 59 .897 2 0.159 9 0.979 1 0.328 6 0. Think of score as likelihood of responding 2.358 5 0.446 4 0.446 4 Comments 1.Data Mining Models Assign a Score to Each Customer ID 0102 0104 0105 0110 0111 0112 0116 0117 0118 Name Will Sue John Lori Beth Pat David Frank Ethel State MA NY AZ AZ NM WY ID MS NE Score Rank 0.265 8 0.314 7 0.

000 customers ACME contacts the highest scoring 300.000 for a direct mail campaign Assumptions each item being mailed costs $1 this cost assumes a minimum order of 20.000 customers Let’s assume ACME is selecting from the top three deciles 60 .000 ACME can afford to contact 300.Approach 1: Budget Optimization ACME has a budget of $300.

we realize “lift” This is a key goal in data mining 61 .The Concept of Lift If we look at a random 10% of the potential customers. we expect to get 10% of likely responders Can we select 10% of the potential customers and get more than 10% of likely responders? If so.

. Response M odel No M odel 3.The Cumulative Gains Chart 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Notes 1. x-axis gives population percentile y-axis gives % captured responses customers with top 10% of the scores account for 30% of responders 62 2.

1. x-axis gives population percentile y-axis gives the lift the top 10% of the scorers are 3 times more likely to respond than a random 10% would be 3 2.5 20% 30% 40% 50% 60% 70% 80% 90% 100% 63 .The Actual Lift Chart 3.5 2 2. 3.5 1 Response M odel No M odel 0 10% 0.5 Notes 1.

but in different ways 64 . 30%) results From lift chart. a response rate of 65% (vs.17 The two charts convey the same information.How Well Does ACME Do? ACME selects customers from the top three deciles From cumulative gains chart. we see a lift of 65/30 = 2.

say 30.000 take note of the 1 to 2% of those who respond build predictive models to predict response use the results from these models The key is to learn from the test marketing campaign 65 .Can ACME Do Better? Test marketing campaign send a mailing to a subset of the customers.

determine the size of the mailing Develop a model to score all customers with respect to their relative likelihood to respond to the offer Choose the appropriate number of top scoring customers 66 .Optimizing the Budget Decide on the budget Based on cost figures.

we need more information than before 67 .Approach 2: Optimizing the Campaign Lift allows us to contact more of the potential responders It is a very useful measure But. how much better off are we financially? We seek a profit-and-loss statement for the campaign To do this.

$1 = $44 68 . warehousing. the net revenue per customer in the campaign is $100 .$55 . $55 covers the cost of inventory. shipping. and so on the cost of sending mail to each customer is $1 Then.Is the Campaign Profitable? Suppose the following the typical customer will purchase about $100 worth of merchandise from the next catalog of the $100.

is predicted to respond ACTUAL YES NO -$1 $0 those who actually respond yield a gain of $45 those who don’t respond yield no gain Predicted Those predicted to respond cost $1 YES NO $44 $0 Those not predicted to respond cost $0 and yield no gain 69 .The Profit/Loss Matrix Someone who scores in the top 30%.

let’s focus on the profit/loss matrix 70 .The Profit/Loss Matrix--continued The profit/loss matrix is a powerful concept But. it has its limitations people who don’t respond become more aware of the brand/product due to the marketing campaign they may respond next time people not contacted might have responded had they been invited For now.

guesswork based on models of customer buying behavior 71 .How Do We Get the P/L Numbers? Cost numbers are relatively easy mailing and printing costs can be handled by accounts payable call center costs. back-of-envelope calculations. for incoming orders. are usually fixed Revenue numbers are rough estimates based on previous experience.

Is the Campaign Profitable? Assumptions made so far $44 net revenue per responder ($1) net revenue per non-responder 300.000 Resulting lift is 2.17 We can now estimate profit for different response rates 72 .000 in target group new assumption: overhead charge of $20.

000 300.000 276.000 300.000 $ (50.000 $ 85.000 The campaign makes money if it achieves a response rate of at least 3% 73 .000 300.000 300.000 $ (185.000 Total Net Revenue 300.46% 0.000 $ 625.000 $ 760.69% 4.62% 9.77% 3.15% 4.030.000) 300.92% Size (YES) 3.38% 1.000 294.000 300.000 27.000 300.000 24.000) 3% 4% 5% 6% 7% 8% 9% 10% 1.000 $ 355.000 300.000 18.000 $ 1.000 30.000 300.23% 3.000 279.000 Size (NO) 297.85% 2.000 $ 220.000 $ 490.000 $ 895.000 270.31% 2.000 273.000 12.000 21.000 282.000 288.000 15.000 6.000 291.000 285.Net Revenue for the Campaign Resp Rate 1% 2% Overall Resp Rate 0.

38% Suppose response rate of 6% 74 .000 = $85.000  (-1) .17 = 1.Net Revenue Table Explained Suppose response rate of 3% Net revenue = 9000  44 + 291.20.000 Lift = response rate for campaign overall response rate  overall response rate = response rate for campaign lift 3% = 2.

. we try to predict a rare event (e.Two Ways to Estimate Response Rates Use a randomly selected hold-out set (the test set) this data is not used to build the model the model’s performance on this set estimates the performance on unseen data Use a hold-out set on oversampled data most data mining involves binary outcomes often.g. fraud) with oversampling. we overrepresent the rare outcomes and underrepresent the common outcomes 75 .

Oversampling Builds Better Models for Rare Events Suppose 99% of records involve no fraud A model that always predicts no fraud will be hard to beat But. such a model is not useful Stratified sampling with two outcomes is called oversampling  76 .

225 80% 90% 100% 2% 0% 100% 100% 1.000 3.343 1.Return to Earlier Model 100% DECILE 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% GAINS 0% 30% CUM 0% 30% LIFT 0.000 0% 10% 20% 30% 40% 50% 60% 70% Response Model No Model 20% 15% 13% 7% 5% 4% 4% 50% 65% 78% 85% 90% 94% 98% 2.000 77 .500 1.500 2.950 1.700 1.167 1.111 1.

000 300.000 SIZE(YES) 3.000 293.000 3.000 2.00% 2.000) Remember: $44 net revenue/ $1 cost per item mailed/ $20.000 6.200 491.00% 94% 98% 100% 100% 1.000 5.200 890.500 SIZE 0 100.343 1.000 PROFIT ($20.111 1.500) ($215.000 $5.000 10.00% 20.00% 7.000) ($470.000) ($137.000) $15.00% 5.700 1.800 10.600 790.00% 30.000 SIZE(NO) 97.500 392.000 200.167 1.000 700.000 195.00% 0.500) ($69.000 ($27.000 800.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECILE 0% 10% 20% 30% 40% 50% 60% GAINS 0.00% 13.400 9.800 8.500 591.950 1.000 400.000 overhead 78 .500 9.00% 4.000) ($570.000 1.000 9.000 900.500 7.000 500.000) ($397.00% 15.000 600.000.00% CUM 0% 30% 50% 65% 78% 85% 90% LIFT 0.000 990.000 690.500 2.000 ($297.225 1.000) 70% 80% 90% 100% 4.

000 = -27.20.size (no) .167 = 6500 100 profit = 286.500 Notice that top 10% yields the maximum profit Mailing to the top three deciles would cost us money 79 .000 .000  2.000 Example: top three deciles (30% row) size (yes) = 300.20.500 .Review of Profit Calculation Key equations size (yes) = size 100  lift profit = 44  size (yes) .293.

000 -200. $20.000 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 80 .000 -600.000 0 -100.000 -500.000) PROFIT by Decile 100.000 -300.000 -400.000 -700.Typical Shape for a Profit Curve ($44. $1.

determine the cumulative profit for each decile Choose all the deciles up to the one with the highest cumulative profit 81 .Approach 2 Summary Estimate cost per contact. and estimated revenue per responder Build a model and estimate response probabilities for each customer Order the customers by their response scores For each decile. calculate the cumulative number of responders and non-responders Using the estimates. overhead.

20 rather than $1? Sensitivity is a serious problem 82 .The Problem with Campaign Optimization Campaign optimization is very sensitive to the underlying assumptions Suppose the response rate is 2% rather than 1%? Suppose the cost of contacting a customer is $1.

346) ($290.000 400.000 900.00% 2.200 PROFIT ($20.000 600.001 988.000 $50.00% 5.281 11.800) ($380.00% 7.200 ($61.00% CUM 0% 30% 50% 65% 78% 85% 90% LIFT 0.054) ($480.343 1.00% 13.801 9.760 11.000) ($134.000 3.00% 20.167 1.000.000 11.719 788.000 2.000 700.000 7.199 390.000 200.000 ($212.111 1.2% and Calculate the Profit for Each Decile DECILE 0% 10% 20% 30% 40% 50% 60% GAINS 0.000 800.000) $42.00% 94% 98% 100% 100% 1.500 2.800 589.000 $31.000 overhead 83 .00% 4.600 6.000) 70% 80% 90% 100% 4.999 12.500 SIZE 0 100.000 688.00% 30.640 489.240 888.000) Remember: $44 net revenue/ $1 cost per item mailed/ $20.225 1.360 10.800 SIZE(NO) 96.000 SIZE(YES) 3.000 1.000 292.400 194.950 1.200 10.054 $1.00% 0.00% 15.Assume an Overall Response Rate of 1.700 1.000 500.000 300.

000 7.000 1.201 6.111 1.000 500.000 800.00% 30.800 692.000 400.200) ($214.000) ($296.00% 15.00% 13.999 8.000 overhead 84 .840 SIZE(NO) 97.700 1.479 792.00% 20.000) ($12.225 SIZE 0 100.036) ($660.167 1.240 6.000 892.200) 90% 100% 2.343 1.8% and Calculate the Profit for Each Decile DECILE 0% 10% 20% 30% 40% 50% 60% 70% 80% GAINS 0.000 ($560.800 7.200 7.000) Remember: $44 net revenue/ $1 cost per item mailed/ $20.799 393.00% CUM 0% 30% 50% 65% 78% 85% 90% 94% 98% LIFT 0.500 1.500 2.000) ($85.00% 7.200 592.Assume an Overall Response Rate of 0.000) ($381.000) ($40.00% 100% 100% 1.000 SIZE(YES) 2.000 700.000 200.00% 4.00% 4.000.000 294.000 900.00% 5.400 4.950 1.564) ($467.000 5.00% 0.160 PROFIT ($20.000 3.001 992.000 2.521 7.000 600.600 196.760 493.964) ($139.000 300.

400 880.00% 2.000 300.090 $282.000 500.000 1.000 $245.000 $230.00% 7.00% 15.000.111 1.000 $190.198 780.500 1.000) Remember: $44 net revenue/ $1 cost per item mailed/ $20.000 681.00% CUM 0% 30% LIFT 0.802 19.500 2.000 overhead 85 .000 $265.00% 50% 65% 78% 85% 2.002 15.00% 13.167 1.000 SIZE(NO) 94.000 700.000 400.000 600.000 60% 70% 80% 90% 100% 5.00% 0.000 $126.000 ($20.000 3.000 582.000 800.600 17.00% 4.090 $62.090) ($120.998 20.000 SIZE 0 100.950 1.000 SIZE(YES) 6.700 200.000) $150.000 10.998 384.400 483.00% 4.000 900.00% 90% 94% 98% 100% 100% 1.600 19.000 190.00% 30.000 18.000 286.000 20% 30% 40% 50% 20.002 980.Assume an Overall Response Rate of 2% and Calculate the Profit for Each Decile DECILE 0% 10% GAINS 0.000 18.000 13.343 1.225 1.000 PROFIT ($20.

80%) PROF (2.000 $200.Dependence on Response Rate ($44.000) ($600. $1. $20.000) ($800.000 $0 ($200.00%) PROF (1.000) ($400.000) $400.000) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 86 PROF (1.20%) PROF (0.00%) .

950 1.500 1.000 PROFIT ($20.200) ($147.000 200.000 500.000 SIZE(NO) 97.00% 5.000 3.00% 15.000 overhead 87 .500 9.600 790.000 690.000) ($4.00% 4.000 SIZE(YES) 3.440) ($235.225 1.00% 7.400) ($34.500 392.00% 20.500 7.800 8.500 591.000 9.000 293.200 890.000) ($86.000 800.00% 0.400 9.00% 30.800 10.000 6.00% 13.200 491.000 700.167 1.000 2.000 400.00% 4.000 SIZE 0 100.000 10.111 1.00% 2.000 1.2 cost per item mailed/ $20.800) ($333.000) ($768.000 195.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECILE 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% GAINS 0.000 300.700 1.200) ($435.000.040) ($648.000) Remember: $44 net revenue/ $1.500 2.000 990.000 5.000 600.343 1.00% CUM 0% 30% 50% 65% 78% 85% 90% 94% 98% 100% 100% LIFT 0.120) ($537.000 900.

500 7.000 $31.00% 0.800 8.000 10.000 2.500 2.500 1.200) ($96.000 690.500 591.500 392.225 1.000) ($372.200 890.000) ($292.000 SIZE 0 100.00% 2.00% 4.00% 4.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECILE 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% GAINS 0.400 9.400 $44.00% 13.000.000 300.000 6.343 1.000 500.00% 7.000) Remember: $44 net revenue/ $0.000 800.00% 20.000) $34.200 491.000 700.800 10.000 SIZE(NO) 97.950 1.200 $9.00% CUM 0% 30% 50% 65% 78% 85% 90% 94% 98% 100% 100% LIFT 0.000 900.00% 30.000 400.000 3.000 overhead 88 .000 195.111 1.00% 5.880) ($220.000 1.440 ($39.000 SIZE(YES) 3.167 1.000 293.000 5.500 9.000 200.600 790.8 cost per item mailed/ $20.800) ($158.000 9.700 1.000 600.000 PROFIT ($20.00% 15.000 990.

00% 90% 94% 98% 100% 100% 1.000) ($461.000 PROFIT ($20.560.200) ($629.500 2.000 293.000) ($987.000 690.800 8.000 400.00% 15.000 300.00% 13.00% 30.000 9.500 7.400 9.600) ($1.167 1.00% CUM 0% 30% LIFT 0.000 990.500 195.00% 4.00% 0.000 ($806.800 10.000) Remember: $44 net revenue/ $2 cost per item mailed/ $20.000) ($82.00% 7.000 SIZE(NO) 97.000 overhead 89 .200 491.000 900.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECIL E 0% 10% GAINS 0.000 6.360.000) ($1.00% 2.169.000 1.000 SIZE 0 100.000 10.200 890.000) 60% 70% 80% 90% 100% 5.600 790.000) ($321.000 3.000 700.000 500.200) ($1.000 9.111 1.000) 20% 30% 40% 50% 20.225 1.000 800.950 1.700 200.343 1.000 600.000 SIZE(YES) 3.500 392.000.000 591.500 1.000 5.00% 50% 65% 78% 85% 2.00% 4.500 ($190.

000) ($600.000) ($1.80) PROF ($2.000.000) ($1.000) ($1.400.000) ($800.00) .000) ($1.Dependence on Costs $200.20) PROF ($0.200.800.000 $0 ($200.000) ($1.000) ($400.600.000) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 90 PROF ($1) PROF ($1.

000) ($11.000 ($39.400 9.000 293.00% 0.000 PROFIT ($20.111 1.167 1.00% 4.000 overhead 91 .225 1.700 1.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECILE 0% 10% GAINS 0.000 600.240) ($558.000 6.000) Remember: $35.00% 2.000 990.500 9.000 3.00% 50% 65% 78% 85% 90% 94% 98% 100% 100% 2.00% 15.500 591.000 800.800 10.000 690.400) 20% 30% 40% 50% 60% 70% 80% 90% 100% 20.800 8.300) ($294.00% 7.2 net revenue/ $1 cost per item mailed/ $20.200) ($379.000 300.500 1.00% 13.500 7.00% 30.000) ($84.000 700.950 1.000 9.000) ($658.000 SIZE(NO) 97.000 900.640) ($212.000 SIZE 0 100.000 1.000.200 890.700) ($137.000 SIZE(YES) 3.000 200.00% CUM 0% 30% LIFT 0.000 195.600 790.000 5.200 491.000 500.500 2.00% 4.00% 5.343 1.500 392.000 10.000 400.720) ($465.

200 890.500 1.000 900.000 400.000) Remember: $52.800 10.760) ($382.00% 15.000 SIZE(YES) 3.280) ($292.111 1.000 300.000 293.00% 30.000 690.000 1.000 $49.800) ($214.00% 7.950 1.000) ($482.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECILE 0% 10% GAINS 0.700) ($135.400 9.600 790.700 ($360) ($62.000 $29.000 200.000 9.343 1.000 overhead 92 .000 195.500 9.00% 4.500 591.000 700.000.000 990.200 491.225 1.000 600.8 net revenue/ $1 cost per item mailed/ $20.500 7.00% CUM 0% 30% LIFT 0.500 392.000) $41.00% 13.000 3.800 8.167 1.000 800.000 SIZE 0 100.000 5.00% 50% 65% 78% 85% 90% 94% 98% 100% 100% 2.00% 2.00% 4.000 500.00% 0.000 SIZE(NO) 97.000 10.700 1.000 PROFIT ($20.500 2.000 6.400 20% 30% 40% 50% 60% 70% 80% 90% 100% 20.00% 5.

000 5.343 1.000 SIZE(NO) 97.500 7.000 10.000) ($130.000 $225.225 1.950 1.000 200.000 195.000 $258.000 SIZE(YES) 3.Assume an Overall Response Rate of 1% and Calculate the Profit for Each Decile DECILE 0% 10% GAINS 0.500 9.000 300.000 700.000 3.00% 30.000 293.00% 13.00% CUM 0% 30% LIFT 0.000 400.400 9.000 $116.000 9.167 1.000 500.000 20% 30% 40% 50% 60% 70% 80% 90% 100% 20.00% 7.600 790.000 PROFIT ($20.200 $236.000 800.200 491.000) $147.00% 15.800 10.500 $181.00% 4.000 690.500 1.111 1.700 1.000.000 990.200 890.200 ($30.00% 5.500 2.000 overhead 93 .000) Remember: $88 net revenue/ $1 cost per item mailed/ $20.000 600.800 8.500 591.500 392.00% 4.00% 0.00% 50% 65% 78% 85% 90% 94% 98% 100% 100% 2.600 $52.000 1.000 900.000 6.000 SIZE 0 100.00% 2.500 $274.

000) ($800.000) ($400.000) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 94 PROF ($44) PROF ($35.Dependence on Revenue $400.20) PROF ($52.000 $200.80) PROF ($88) .000) ($600.000 $0 ($200.

our profit estimates could be off by a lot In addition. cost estimates. the same group of customers is chosen for multiple campaigns 95 .Campaign Optimization Drawbacks Profitability depends on response rates. and revenue potential Each one impacts profitability The numbers we use are just estimates If we are off by a little here and a little there.

Approach 3: Customer Optimization Campaign optimization makes a lot of sense But. rather than campaigns 96 . campaign profitability is difficult to estimate Is there a better way? Do what is best for each customer Focus on customers.

etc. etc. insurance: home. etc.Real-World Campaigns Companies usually have several products that they want to sell telecom: local. mortgages. mobile. personal liability. banking: CDs. credit cards. long distance. ISP. car. retail: different product lines There are also upsell and customer retention programs These campaigns compete for customers 97 .

if the product and customer are incompatible 1.Each Campaign May Have a Separate Model These models produce scores The score tells us how likely a given customer is to respond to that specific campaign 0. each with a separate data mining model 98 . if the customer has asked about the product Each campaign is relevant for a subset of all the customers Imagine three marketing campaigns. if the customer already has the product 0.

Sample Scores (as Rankings for Three Different Campaigns) ID 0102 0104 0105 0110 0111 0112 0116 0117 0118 Name Will Sue John Lori Beth Pat David Frank Ethel State Mod A MA 3 NY 1 AZ 2 AZ 5 NM 9 WY 4 ID 6 MS 8 NE 6 Mod B 4 2 1 7 3 5 5 9 8 Mod C 2 4 1 6 8 2 7 8 5 99 .

Choose the Best Customers for Each Campaign ID 0102 0104 0105 0110 0111 0112 0116 0117 0118 Name Will Sue John Lori Beth Pat David Frank Ethel State MA NY AZ AZ NM WY ID MS NE Mod A Mod B Mod C 3 4 2 1 2 4 2 1 1 5 7 6 9 3 8 4 5 2 6 5 7 8 9 8 6 8 5 100 .

A Common Situation “Good” customers are typically targeted by many campaigns Many other customers are not chosen for any campaigns Good customers who become inundated with contacts become less likely to respond at all Let the campaigns compete for customers 101 .

Choose the Best Campaign for Each Customer ID 0102 0104 0105 0110 0111 0112 0116 0117 0118 Name Will Sue John Lori Beth Pat David Frank Ethel State MA NY AZ AZ NM WY ID MS NE Mod A Mod B Mod C 3 4 2 1 2 4 2 1 1 5 7 6 9 3 8 4 5 2 6 5 7 8 9 8 6 8 5 102 .

Focus on the Customer Determine the propensity of each customer to respond to each campaign Estimate the net revenue for each customer from each campaign Incorporate profitability into the customeroptimization strategy Not all campaigns will apply to all customers 103 .

0% 3.0% 3.9% 1.7% 11.8% 0.8% 4.8% 0. Determine Response Rate for Each Campaign ID 102 104 105 110 111 112 116 117 Name Will Sue John Lori Beth Pat David Frank State MA NY AZ AZ NM WY ID MS Mod A 2.2% 0.6% 1.0% 1.1% 2.2% Mod C 0.0% 1.8% 0.3% 7.8% 118 Ethel NE 0.2% 0.0% 8.0% Customers who are not candidates are given a rate of zero104 .2% Mod B 0.4% 2.8% 0.1% 0.First.9% 0.0% 0.3% 0.8% 0.

0% 1.8% 0.8% 4.8% 0.0% 0.6% 1.0% $56 $72 $20 104 105 110 111 112 116 117 Sue John Lori Beth Pat David Frank NY AZ AZ NM WY ID MS 8.8% 0.7% 11.2% 0.1% 0.2% 3.4% 2.Second.3% 0.0% $56 $72 $20 As a more sophisticated alternative.9% 0.8% $56 $56 $56 $56 $56 $56 $56 $72 $72 $72 $72 $72 $72 $72 $20 $20 $20 $20 $20 $20 $20 118 Ethel NE 0.2% 0. Add in Product Profitability ID 102 Name State Will MA Mod A 2.2% 1.0% 1.0% Mod B 0.9% Mod C Prof A Prof B Prof C 0.8% 0. profit could be estimated for each customer/product combination 105 .0% 3.3% 7.1% 2.8% 0.

16 $0.92 $0.14 $0. Determine the Campaign with the Highest Value ID Name State EP (A) EP (B) EP (C) Campaign 102 104 105 110 Will Sue John Lori MA NY AZ AZ $1.45 $0.01 $1.22 $0.00 $0.45 $0.13 $0.12 $4.06 $1.48 $2.66 $5.00 B A B C A EP (k) = the expected profit of product k For each customer.04 $0.14 $0.86 $0.74 $2.11 $0.65 $1.58 $0.58 $ 0. choose the highest expected profit campaign 106 .Finally.26 A A C B 111 112 116 117 118 Beth Pat David Frank Ethel NM WY ID MS NE $0.16 $0.12 $0.50 $0.20 $0.

Conflict Resolution with Multiple Campaigns Managing many campaigns at the same time is complex for technical and political reasons Who owns the customer? Handling constraints each campaign is appropriate for a subset of customers each campaign has a minimum and maximum number of contacts each campaign seeks a target response rate new campaigns emerge over time 107 .

Marketing Campaigns and CRM The simplest approach is to optimize the budget using the rankings that models produce Campaign optimization determines the most profitable subset of customers for a given campaign. but it is sensitive to assumptions Customer optimization is more sophisticated It chooses the most profitable campaign for each customer 108 .

The Data Mining Process What role does data mining play within an organization? How does one do data mining correctly? The SEMMA Process select and sample explore modify model assess 109 .

or a real-time response (call centers and web)? What would the ideal result look like? how would it be used? 110 . an on-going monthly batch job. not technical expertise Define the problem clearly “predict the likelihood of churn in the next month for our 10% most valuable customers” Define the solution clearly is this a one-time job.Identify the Right Business Problem Involve the business users Have them provide business expertise.

big enough to contain significant information.Transforming the Data into Actionable Information Select and sample by extracting a portion of a large data set-. but small enough to manipulate quickly Explore by searching for unanticipated trends and anomalies in order to gain understanding 111 .

Transforming the Data into Actionable Information-- continued
Modify by creating, selecting, and transforming the variables to focus the model selection process
Model by allowing the software to search automatically for a combination of variables that reliably predicts a desired outcome Assess by evaluating the usefulness and reliability of the findings from the data mining process
112

Act on Results
Marketing/retention campaign
Personalized messages Customized user experience Customer prioritization Increased understanding of customers, products, messages

 lists or scores

113

Measure the Results
Confusion matrix Cumulative gains chart

Lift chart
Estimated profit

114

Data Mining Uses Data from the Past to Effect Future Action
“Those who do not remember the past are condemned to repeat it.” – George Santayana Analyze available data (from the past) Discover patterns, facts, and associations Apply this knowledge to future actions

115

Examples
Prediction uses data from the past to make predictions about future events (“likelihoods” and “probabilities”) Profiling characterizes past events and assumes that the future is similar to the past (“similarities”) Description and visualization find patterns in past data and assume that the future is similar to the past
116

We Want a Stable Model
A stable model works (nearly) as well on unseen data as on the data used to build it
Stability is more important than raw performance for most applications
we want a car that performs well on real roads, not just on test tracks

Stability is a constant challenge

117

g. what we know about the past may not be relevant to tomorrow users of the web have changed since late 1990s Are the data mining models created from past data relevant to the future? have critical assumptions changed? 118 . demographic data Is the business environment from the past relevant to the future? in the ecommerce era..Is the Past Relevant? Does past data contain the important business drivers? e.

and it produces one or more outputs Sometimes.Data Mining is about Creating Models A model takes a number of inputs. which often come from databases. the purpose is to build the best model The best model yields the most accurate output Such a model may be viewed as a black box Sometimes. the purpose is to better understand what is happening This model is more like a gray box 119 .

Models Past Present Future Data ends here Actions take place here Building models takes place in the present using data from the past outcomes are already known Applying (or scoring) models takes place in the present Acting on the results takes place in the future outcomes are not known 120 .

the Purpose is to Assign a Score to Each Customer ID 1 2 3 4 5 6 0102 0104 0105 0110 0111 0112 Name Will Sue John Lori Beth Pat State Score MA NY AZ AZ NM WY 0. Some scores may be the same 3.446 0.328 Comments 1.159 0.446 121 .358 0.314 0.979 0.Often. Scores are assigned to rows using models 2.265 0.897 0. The scores may represent the probability of some outcome 7 8 9 0116 0117 0118 David Frank Ethel ID MS NE 0.

Common Examples of What a Score Could Mean Likelihood to respond to an offer Which product to offer next Estimate of customer lifetime Likelihood of voluntary churn Likelihood of forced churn Which segment a customer belongs to Similarity to some customer profile Which channel is the best way to reach the customer 122 .

446 0.159 123 0116 David 0116 David .446 0.328 0.446 SORT 8 7 9 4 6 1 3 2 0117 0118 0110 0112 0102 0105 0104 Frank Ethel Lori Pat Will John Sue MS ID NE AZ WY MA AZ NY 0.265 0.The Scores Provide a Ranking of the Customers ID 1 0102 Name State Will MA Score 0.358 0.358 0.897 0.897 0.265 0.328 0.159 0.446 0.314 0.979 0.979 2 3 4 5 6 7 8 9 0104 0105 0110 0111 0112 0117 0118 Sue John Lori Beth Pat Frank Ethel NY AZ AZ NM WY ID MS NE 0.314 5 ID 0111 Name State Beth NM Score 0.

This Ranking give Rise to Quantiles (terciles.979 0.358 0.897 0.446 0. quintiles.314 0. deciles.) ID 0111 0117 0116 0118 0110 0112 0102 0105 0104 Name Beth Frank David Ethel Lori Pat Will John Sue State NM MS ID NE AZ WY MA AZ NY Score 0.328 0.265 0. etc.159 5 8 7 9 4 6 1 3 2 } } } high medium low 124 .446 0.

at bottom of pyramid Business rules tell us what we’ve learned from the data at top of pyramid Other layers in between 125 .Layers of Data Abstraction SEMMA starts with data There are many different levels of data within an organization Think of a pyramid The most abundant source is operational data every transaction. bill. etc. payment.

SEMMA: Select and Sample What data is available? Where does it come from? How often is it updated? When is it available? How recent is it? Is internal data sufficient? How much history is needed? 126 .

Data Mining Prefers Customer Signatures Often. the data come from many different sources Relational database technology allows us to construct a customer signature from these multiple sources The customer signature includes all the columns that describe a particular customer the primary key is a customer id the target columns contain the data we want to know more about (e.. predict) the other columns are input columns 127 .g.

. gender.Profiling is a Powerful Tool Profiling involves finding patterns from the past and assuming they will remain valid The most common approach is via surveys Surveys tell us what our customers and prospects look like Typical profiling question: What do churners look like? Profiling is frequently based on demographic variables e. location. age 128 .g.

Profiling has its Limitations Even at its best. profiling tells us about the past Connection between cause and effect is sometimes unclear people with brokerage accounts have a minimal balance in their savings account customers who churn are those who have not used their telephones (credit cards) for the past month customers who use voicemail make a lot of short calls to the same number More appropriate for advertising than one-to-one marketing 129 .

Two Ways to Aim for the Target Profiling: What do churners look like? data in input columns can be from the same time period (the past) as the target Prediction: Build a model that predicts who will churn next month data from input columns must happen before the target data comes from the past the present is when new data are scored 130 .

The Past Needs to Mimic the Present Past Present Future Distant Past ends here Recent Past starts here Data ends here Predictions start here We mimic the present by using the distant past to predict the recent past 131 .

How Data from Different Time Periods are Used Jan Feb Model Set Score Set Mar Apr May Jun Jul Aug Sep 4 3 2 4 1 3 2 +1 1 +1 The model set is used to build the model The score set is used to make predictions It is now August X marks the month of latency Numbers to left of X are months in the past 132 .

Multiple Time Windows Help the Models Do Well in Predicting the Future Jan Feb Mar Apr May Jun Jul Aug Sep Model Set 4 3 4 2 3 1 2 4 1 3 +1 +1 2 1 +1 Score Set Multiple time windows capture a wider variety of past behavior They prevent us from memorizing a particular season 133 .

Rules for Building a Model Set for a Prediction All input columns must come strictly before the target There should be a period of “latency” corresponding to the time needed to gather the data The model set should contain multiple time windows of data 134 .

More about the Model and Score Sets The model set can be partitioned into three subsets the model is trained using pre-classified data called the training set the model is refined. using the test set the performance of models can be compared using a third subset called the evaluation or validation set The model is applied to the score set to predict the (unknown) future 135 . in order to prevent memorization.

Stability Challenge: Memorizing the Training Set
Error Rate

Training Data

Model Complexity

Decision trees and neural networks can memorize nearly any pattern in the training set
136

Danger: Overfitting
This is the model we want

Error Rate

The model has overfit the training data As model complexity grows, performance deteriorates on test data Model Complexity
137

Building the Model from Data
Both the training set and the test set are used to create the model Algorithms find all the patterns in the training set
some patterns are global (should be true on unseen data) some patterns are local (only found in the training set)

We use the test set to distinguish between the global patterns and the local patterns

Finally, the validation set is needed to evaluate the model’s performance
138

SEMMA: Explore the Data Look at the range and distribution of all the variables Identify outliers and most common values Use histograms, scatter plots, and subsets Use algorithms such as clustering and market basket analysis

Clementine does some of this for you when you load the data
139

SEMMA: Modify
Add derived variables
total, percentages, normalized ranges, and so on extract features from strings and codes

Add derived summary variables
median income in ZIP code

Remove unique, highly skewed, and correlated variables
often replacing them with derived variables

Modify the model set
140

The Density Problem
The model set contains a target variable
“fraud” vs. “not fraud” “churn” vs. “still a customer”

Often binary, but not always The density is the proportion of records with the given property (often quite low)
fraud ≈ 1% churn ≈ 5%

Predicting the common outcome is accurate, but not helpful
141

5 142 .Back to Oversampling 1 1 1 2 1 3 1 4 1 2 1 2 2 2 3 2 4 2 3 1 3 2 3 3 3 4 3 4 1 4 2 4 3 4 4 4 5 1 5 2 5 3 5 4 5 6 1 6 2 6 3 6 4 6 7 1 7 2 7 3 7 4 7 8 1 8 2 8 3 7 4 8 9 1 9 2 9 3 9 4 9 1 0 2 0 3 0 4 0 5 0 2 1 2 2 3 3 1 4 5 6 1 7 2 9 3 4 4 8 1 0 2 0 3 0 4 0 5 0 Original data has 45 white and 5 dark (10% density) The model set has 10 white and 5 dark (33% density ) For every 9 white (majority) records in the original data. two are in the oversampled model set Oversampling rate is 9/2 = 4.

aim for at least 10. 50% for binary outcomes 143 .Two Approaches to Oversampling Build a new model set of the desired density fewer rows takes less time to build models more time for experimentation in practice.000 rows Use frequencies to reduce the importance of some rows uses all of the data Use a density of approx.

Oversampling by Taking a Subset of the Model Set ID 1 2 3 4 0102 0104 0105 0110 Name Will Sue John Lori State MA NY AZ AZ Flag F F F F ID Name State Flag 1 3 5 6 8 9 0102 0105 0111 0112 0117 0118 Will John Beth Pat Frank Ethel MA AZ NM WY MS NE F F T F T F 5 6 7 8 9 0111 0112 0116 0117 0118 Beth Pat David Frank Ethel NM WY ID MS NE T F F T F The original data has 2 Ts and 7 Fs (22% density) Take all the Ts and 4 of the Fs (33% density) The oversampling rate is 7/4 = 1.75 144 .

Oversampling via Frequencies
ID 1 2 0102 0104 Name Will Sue State MA NY Flag F F 1 2 ID 0102 0104 Name Will Sue State MA NY Flag F F Frq 0.5 0.5

3
4 5 6

0105
0110 0111 0112

John
Lori Beth Pat

AZ
AZ NM WY

F
F T F

3
4 5 6

0105
0110 0111 0112 0117 0118

John
Lori Beth Pat Frank Ethel

AZ
AZ NM WY

F
F T F

0.5
0.5 1.0 0.5

7
8 9

0116
0117 0118

David
Frank Ethel

ID
MS NE

F
T F

7
8 9

0116 David

ID
MS NE

F
T F

0.5
1.0 0.5

Add a frequency or weight column for each F, Frq = 0.5 for each T, Frq = 1.0 The model set has density of 2/(2 + 0.5  7) = 36.4% The oversampling rate is 7/3.5 = 2
145

SEMMA: Model
Choose an appropriate technique
decision trees neural networks regression combination of above

Set parameters

Combine models

146

Regression
Tries to fit data points to a known curve
(often a straight line) Standard (well-understood) statistical technique Not a universal approximator (form of the regression needs to be specified in advance)

147

Neural Networks
Based loosely on computer models of how brains work Consist of neurons (nodes) and arcs, linked together

Each neuron applies a nonlinear function to its inputs to produce an output
Particularly good at producing numeric outputs No explanation of result is provided
148

Decision Trees
Looks like a game of “Twenty Questions” At each node, we fork based on variables
e.g., is household income less than $40,000?

These nodes and forks form a tree

Decision trees are useful for classification problems
especially with two outcomes

Decision trees explain their result
the most important variables are revealed
149

Experiment to Find the Best Model for Your Data
Try different modeling techniques Try oversampling at different rates Tweak the parameters Add derived variables Remember to focus on the business problem

150

502 0.615 7 8 9 0116 0117 0118 David Frank Ethel ID MS NE 0.121 0.133 Mod 2 0.265 Mod 3 0.510 0.411 0.146 0.323 0.211 4 5 6 0110 0111 0112 Lori Beth Pat AZ NM WY 0.105 0.It is Often Worthwhile to Combine the Results from Multiple Models ID 1 2 3 0102 0104 0105 Name Will Sue John State MA NY AZ Mod 1 0.879 0.152 0.159 0.111 0.925 0.419 0.314 0.116 0.893 0.611 151 .491 0.979 0.692 0.358 0.446 0.298 0.

Multiple-Model Voting Multiple models are built using the same input data Then a vote. is used for the final classification Requires that models be compatible Tends to be robust and can return better results 152 . often a simple majority or plurality rules vote.

Segmented Input Models Segment the input data by customer segment by recency Build a separate model for each segment Requires that model results be compatible Allows different models to focus and different models to use richer data 153 .

best customers are not always most frequent Thus. two models were developed who will respond? how much will they give? 154 . the less money one contributes each time so.Combining Models What is response to a mailing from a non-profit raising money (1998 data set) Exploring the data revealed the more often.

Compatible Model Results In general. the probability depends on the density of the model set The density of the model set depends on the oversampling rate 155 . the score may be interpreted as the probability of an outcome However. the score refers to a probability for decision trees. the score may be the actual density of a leaf node for a neural network.

An Example The original data has 10% density The model set has 33% density Each white in model set represents 4.5 156 .5 white in original data Each dark represents one dark The oversampling rate is 4.

versus the density of 33% for the entire model set This score represents the probability on the oversampled data This group has a lift of 67/33 = 2 157 .A Score Represents a Portion of the Model Set Suppose an algorithm identifies the group at right as most likely to churn The score would be 4/6 = 67%.

Determining the Score on the Original Data The corresponding group in the original data has 4 dark and 9 white.7/10 = 3.7% The original data has a density of 10% The lift is now 30. for a score of 4 / (4 + 9) = 30.07 158 .

it corresponds to 13/50 = 26% Bottom line: before comparing the scores that different models produce.continued The original group accounted for 6/15 = 40% of the model set In the original data.Determining the Score -. make sure that these scores are adjusted for the oversampling rate The final part of the SEMMA process is to assess the results 159 .

it is right 800/850 = 94% of the time Predicted Yes 800 No 50 100 When the model predicts No. it is right 100/150 = 67% of the time The density of the model set is 150/1000 = 15% 160 .Confusion Matrix (or Correct Classification Matrix) Actual Yes No 50 There are 1000 records in the model set When the model predicts Yes.

continued The model is correct 800 times in predicting Yes The model is correct 100 times in predicting No The model is wrong 100 times in total The overall prediction accuracy is 900/1000 = 90% 161 .Confusion Matrix-.

265 0.From Data to the Confusion Matrix ID Name State MA NY AZ AZ NM WY ID Score 0.446 T F T T We hold back a portion of the data so we have scores and actual values The top tercile is given a predicted value of T Because of tie.358 0.314 0.979 0. we have 4 Ts predicted 162 .897 0.446 Act F F F T T F F Pred F F F F T F T 1 0102 Will 2 0104 Sue 3 0105 John 4 0110 Lori 5 0111 Beth 6 0112 Pat 7 0116 David Actual T Predicted T F 2 1 F 2 4 8 0117 Frank 9 0118 Ethel MS NE 0.328 0.159 0.

How Oversampling Affects the Results Actual Yes No Yes Actual No 50 100 Predicted Predicted Yes 800 50 Yes No 8000 No 50 100 500 Model set Original data The model set has a density of 15% No Suppose we achieve this density with an oversampling rate of 10 So. for every Yes in the model set there are 10 Yes’s in the original data 163 .

7% The results will vary based upon the degree of oversampling 164 .How Oversampling Affects the Results-continued Original data has a density of 150/8650 = 1.7% of the time The accuracy has gone down from 67% to 16.734% We expect the model to predict No correctly 100/600 = 16.

3% The density of dark in the subset chosen by the model is 66.7% The lift is 66.3 = 2 The model is doing twice as well as choosing circles at random 165 .7/33.Lift Measures How Well the Model is Doing The density of dark in the model set is 33.

358 0.7% The lift is 66.979 0.7/33.328 0.159 0.265 0.446 0.Lift on a Small Data Set ID Name State MA NY AZ AZ NM WY ID MS NE Score 0.3% Tercile 1 has two T and one F Tercile 1 has a density of 66.3 = 2 166 .446 Act F F F T T F F T F Pred F F F F T F T T T Tercile 3 3 3 2 1 2 1 1 2 1 0102 Will 2 0104 Sue 3 0105 John 4 0110 Lori 5 0111 Beth 6 0112 Pat 7 0116 David 8 0117 Frank 9 0118 Ethel Note: we break tie arbitrarily Model set density of T is 33.897 0.314 0.

0 1. 3 1 2 and 3 comprise the Tercile entire model set.3 = 2 Terciles 1 and 2 have a density of 3/6 = 50% and a lift of 50/33. 2.5 Since terciles 1. the lift is 1 We always look at lift in a cumulative sense 167 .5 1.7/33.The Lift Chart for the Small Data Set 2.0 0.5 Tercile 1 has a lift of 66.3 = 1.5 2.

Cumulative Gains Chart 100% * * 67% * * Cumulative gains chart shows the proportion of responders (churners) in each tercile (decile) Horizontal axis shows the tercile (decile) 33% * Vertical axis gives the proportion of responders that model yields 67% 100% 33% The cumulative gains chart and the lift chart are related 168 .

More on Lift and the Cumulative Gains Chart The lift curve is the ratio of the cumulative gains of the model to the cumulative gains of the random model The cumulative gains chart has several advantages it always goes up and to the right can be used diagnostically Reporting the lift or cumulative gains on the training set is cheating Reporting the lift or cumulative gains without the oversampling rate is also cheating 169 .

Summary of Data Mining Process Cycle of Data Mining identify the right business problem transform data into actionable information act on the information measure the effect SEMMA select/sample data to create the model set explore the data modify data as necessary model to produce results assess effectiveness of the models 170 .

The Data in Data Mining Data comes in many forms internal and external sources Different sources of data have different peculiarities Data mining algorithms rely on a customer signature one row per customer multiple columns See Chapter 6 in Mastering Data Mining for details 171 .

Preventing Customer Attrition We use the noun churn as a synonym for attrition We use the verb churn as a synonym for leave Why study attrition? it is a well-defined problem it has a clear business value we know our customers and which ones are valuable we can rely on internal data the problem is well-suited to predictive modeling 172 .

especially if they are costing money Don’t intervene in every case Topic should be called “managing customer attrition” 173 .When You Know Who is Likely to Leave. You Can … Focus on keeping high-value customers Focus on keeping high-potential customers Allow low-potential customers to leave.

The Challenge of Defining Customer Value We know how customers behaved in the past We know how similar customers behaved in the past Customers have control over revenues. but much less control over costs we may prefer to focus on net revenue rather than profit We can use all of this information to estimate customer worth These estimates make sense for the near future 174 .

customer attrition becomes more important total customers new customers churners 9 10 11 175 Year .Why Maturing Industries Care About Customer Attrition Representative Growth in a Maturing Market Number of Customers (1000s) 1200 1000 800 600 400 200 0 0 1 2 3 4 5 6 7 8 In this region of rapid growth. building infrastructure is more important than CRM As growth flattens out.

customer attrition and customer profitability become issues Time Period In a mature market. the amount of attrition almost equals the number of new customers 176 . almost all customers are new customers In a maturing market.Another View of Customer Attrition Num ber of Churners per 100 New Custom ers 100 90 80 70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 In a fast growth market.

it is cheaper to prevent attrition than to spend money on customer acquisition One reason is that. acquisition costs go up 177 . attrition is usually not important customer acquisition is more important Eventually. every new customer is simply replacing one who left Before this point.Why Attrition is Important When markets are growing rapidly. as a market becomes saturated.

$20 offer 120 70 53.3 45 40 1.00% 2. Acquisition Response Rates Decrease and Costs Increase $250 Cost per Response $200 $150 $100 $50 $0 0.00% 5.00% 220 Assumption: $1 per contact.00% 6.00% 4.00% 3.00% Response Rate 178 .In Maturing Markets.

00% 4.00% 3. $20 offer 120 70 53.00% Response Rate 179 .00% 2.3 45 40 1.Acquisition versus Retention $250 Cost per Response $200 $150 $100 $50 $0 0.00% 220 Assumption: $1 per contact.00% 5.00% 6.

continued As response rate drops. we could spend $140 to retain an existing customer Assume the two customers have the same potential value Some possible options decrease the costs of the acquisition campaign implement a customer retention campaign combination of both 180 .Acquisition versus Retention-. suppose we spend $140 to obtain a new customer Alternatively.

but some would have remained customers anyway cost of retention is the campaign cost divided by the net increase in retained customers 181 .Retention and Acquisition are Different Economics of acquisition campaigns customers are unlikely to purchase unless contacted/invited cost of acquisition is the campaign cost divided by the number acquired during the campaign Economics of retention campaigns customers are targeted for retention.

response rates tend to be high As before. we need to distinguish between customers who merely respond and those who respond and would have left If the overall group has an attrition rate of 25%. assume $1 per contact and $20 per response We need to specify response rate and attrition rate 182 .Cost per Retention: How Many Responders Would Have Left? For a retention campaign. you can assume that one quarter of the responders are saved Since people like to receive something for nothing.

0% 16.0% 2.0% 8.00% $1.00% 10.0% 18.0% 22.0% 10.0% 4.0% 14.0% 20.00% Cost Per Retention $1.400 5.200 15.00% 25.0% Response Rate of Compaign 183 .600 $1.0% 6.0% 12.00% $600 $400 $200 $0 0.00% 20.Cost Per Retention by Attrition Rate $1.000 $800 35.

but are retained campaign cost = $100 + $20  20 = $500 cost per retention = $500/2 = $250 184 . $20 per response What is the cost per retention with response rate of 20% and attrition rate of 10%? Suppose 100 people are contacted 20 people respond 20  10% = 2 would have left.A Sample Calculation Given: $1 per contact.

994 326.880 114.485 815. the number of customers is the number at the end of the previous year minus the number who left plus the number of new customers 185 .000 118.000 52.300 153. 40% of existing customers leave Each year.610 40.905 337. 70% new customers are added At year end.731 148.450 Each year.700 285.000 219.681 627.072 250.485 815.790 199.000 91.244 70.517 193.Typical Customer Retention Data Num Customers Beg of Year Num Churners 40% 0 0 New Customers 70% 100.700 285.731 1.000 169.000 130.610 371.293 482.060.681 627.927 130.000 Num Customers End of Year 1990 1991 1992 1993 1994 100.600 87.000 169.877 439.293 1995 1996 1997 1998 371.292 259.000 67.012 482.000 219.240 571.

485 815.450 30.77% 30.517 193.293 482.292 New Customers 70% 100.012 0 130.000 91.000 70.240 571.77% 30.681 627.000 169.000 219.994 326.000 169. we reduce the attrition rate to about 31% per year This may look better.877 439.77% 30.700 285.927 259.000 52.77% 30.How to Lie with Statistics Num Customers Beg of Year Num Churners 40% 0 1990 1991 1992 1993 1994 1995 1996 1997 1998 100.072 250.000 67.293 482.485 815.77% Num Customers End of Year Reported Churn When we divide by end-of-year customers.731 1.905 337.77% 30.731 0 40.244 148.600 87.610 371.000 130.77% 30.060.000 118.77% 30. but it is cheating 186 .610 371.77% 30.000 219.300 153.880 114.790 199.681 627.700 285.

77% 30.77% 30.000 219.731 1.293 482.000 130.060.517 193.012 0 482.700 285.700 285.000 52.77% 30.77% 30.610 40.000 219.072 250.270 30.880 114.000 118.905 337.681 627.600 87.994 326.790 199.681 627.300 153.000 169.244 70.77% 30.000 0 Num Customers End of Year Reported Churn 1990 1991 1992 1993 1994 100.610 371.77% 1995 1996 1997 1998 1999 371.67% If acquisition of new customers stops.485 815.Suppose Acquisition Suddenly Stops Num Customers Beg of Year Num Churners 40% 0 0 New Customers 70% 100.77% 66.77% 30.240 571.485 815.450 148.927 130.731 1.292 424.000 169.877 439.77% 30. churn rate more than doubles 187 .000 91.450 636.000 67. existing customers still leave at same rate But.293 30.180 259.060.

we assume that new customers do not leave during their first year the real world is more complicated we might look at the number of customers on a monthly basis 188 .Measuring Attrition the Right Way is Difficult The “right way” gives a value of 40% instead of 30.77% who wants to increase their attrition rate? this happens when the number of customers is increasing For our small example.

000.000 $0 -$5.000 $20.000.000 90 91 92 93 94 95 96 97 98 Lo s s D u e t o A t t r i t i o n Revenue Time Assumption: $20 for existing customers. $10 for new customers 189 .000.000.Effect of Attrition on Revenue Effect of Attrition on Revenue $25.000 $10.000.000.000 Ne w C u s t o m e r R e v e n u e Ex i s t i n g C u s t o m e r R e v e n u e $15.000 $5.

the loss due to attrition is about the same as the new customers revenue The loss due to attrition has a cumulative impact If we could retain some of these customers each year.continued From previous page.Effect of Attrition on Revenues -. attrition is greater for longer-duration customers 190 . they would generate revenue well into the future It is useful to examine the relationship between attrition and the length of the customer relationship Often.

Relationship Between Attrition and the Length of the Customer Relationship 500.000 350.000 200.000 150.000 250.000 100.000 50.000 Num Customers Current Customers Churners 300.000 0 0 1 2 3 4 5 6 7 8 Length of Relationship 191 .000 450.000 400.

What is a Customer Attrition Score? When building an attrition model. we seek a score for each customer This adds a new column to the data Two common approaches a relative score or ranking of who is going to leave an estimate of the likelihood of leaving in the next time period 192 .

it does allow for more targeted marketing to customers it is often possible to gain understanding and combat attrition while using attrition models 193 .Attrition Scores are an Important Part of the Solution Having an idea about which customers will leave does not address key business issues why are they leaving? where are they going? is brand strength weakening? are the products still competitive? However.

Requirements of Effective Attrition Management Keeping track of the attrition rate over time Understanding how different methods of acquisition impact attrition Looking at some measure of customer value. to determine which customers to let leave Implementing attrition retention efforts for high-value customers Knowing who might leave in the near future 194 .

Three Types of Attrition Voluntary attrition when the customer goes to a competitor our primary focus is on voluntary attrition models Forced attrition when the company decides that it no longer wants the customer and cancels the account often due to non-payment our secondary focus is on forced attrition models Expected attrition when the customer changes lifestyle and no longer needs your product or service 195 .

What is Attrition? In the telecommunications industry it is very clear customers pay each month for service customers must explicitly cancel service the customer relationship is primarily built around a single product It is not so clear in other industries retail banking credit cards retail e-commerce 196 .

Consider Retail Banking Customers may have a variety of accounts deposit and checking accounts savings and investment mortgages loans credit cards business accounts What defines attrition? closing one account? closing all accounts? something else? 197 .

Retail Banking--Continued One large bank uses checking account balance to define churn What if a customer (e. credit card or mortgage customer) does not have a checking account with the bank? Another example from retail banking involves forced attrition Consider the bank loan timeline on the next page 198 ..g.

The Path to Forced Attrition Good Customer Missed Payment 30 Days Late 60 Days Late Forced Attrition: the bank sells the loan to a collection agency 199 90 Days Late .

in the U.Credit Cards Credit cards. typically don’t have an annual fee there is no incentive for a cardholder to tell you when she no longer plans on using the card Cardholders typically use the card several times each month Customers who stop using the card and are not carrying a balance are considered to be silent churners The definition of silent churn varies among issuers customers who have not used the card for six months customers who have not used the card for three months and for nine of the last twelve months 200 ..S.

The Catalog Industry Like credit cards. customers have little incentive to cancel them Unlike credit cards. catalogs are free therefore. purchases from catalogs are more sporadic so defining silent churn is more difficult Purchases from catalogs are often seasonal so silent churn is measured over the course of years. rather than months 201 .

so there is no incentive to announce intention to churn Customers often visit many times before making a purchase Some customers buy frequently and others don’t how can Amazon decrease time between purchases? The e-commerce world is still growing rapidly. so customer retention is not a major issue But it will become a major issue soon 202 .E-commerce Attrition in e-commerce is hardest of all to define consider Amazon Web sites are free.

discounts 203 . free weekend airtime.Ways to Address Voluntary Attrition Allow customers who are not valuable to leave Offer incentives to stay around for a period of time teaser rates on credit cards. no payments or interest for six months Offer incentives to valuable customers discounts/extras Stimulate usage miles for minutes. donations to charities.

again.Predicting Voluntary Attrition Can Be Dangerous Voluntary attrition sometimes looks like expected or forced attrition Confusing voluntary and expected attrition results in the waste of marketing dollars Confusing voluntary and forced attrition means you lose twice by. wasting marketing dollars by incurring customer non-payment 204 .

billing inserts.Ways to Address Forced Attrition Stop marketing to the customer no more catalogs. or other usage stimulation Reduce credit lines Increase minimum payments Accelerate cancellation timeline Increase interest rates to increase customer value while he is still paying 205 .

Predicting Forced Attrition Can Be Dangerous Forced attrition sometimes looks at good customers they have high balances they are habitually late Confusing forced attrition with good customers may encourage them to leave Confusing forced with voluntary attrition may hamper winback prospects 206 .

Attrition Scores are Designed for the Near Future The chance that someone will leave tomorrow is essentially 0% The chance that someone will leave in the next 100 years is essentially 100% A good attrition score is valid for one to three months in the future Attrition modeling can take place at any point in the customer lifecycle 207 .

we know when he will churn This can help us obtain an estimate of customer profitability Customers with a high lifetime value (profitability) are the ones we want to prioritize for retention efforts 208 .A Related Problem: Estimating the Customer Lifetime We also want to estimate the length of a customer’s lifetime If we know how long someone will remain a customer.

9 .1 209 .9)x1 .Suppose Customers Have a 10% Chance of Leaving in the Next Month Probability the customer leaves in next month = .1% have a lifetime of 3 months (round up) Probability that customer leaves in x months = (.1  10% of customers have a lifetime of 1 month (round up) Probability the customer leaves in 2 months = .081  8.1= .1= .09  9% have a lifetime of 2 months (round up) Probability the customer leaves in 3 months = .9 .9 .

the average lifetime is 1/x% months In this example.1= 10 months An ideal attrition model says all customers leaving in the next month get a score of 1 or 100% all other customers get a score of 0 or 0% 210 . we refer to this as a geometric distribution If the churn rate per month is x%. the average lifetime is 1/.Average Customer Lifetime In statistics.

forced. expected Related to attrition is the customer lifetime useful in calculating lifetime customer value 211 .Attrition Modeling Summary It is very important to manage attrition. especially in maturing industries Slowing attrition may be the cheapest way to maintain a critical mass of customers There are different types of attrition voluntary.

adapted. in preparing these slides 212 . I have also consulted Data Mining Techniques (Wiley. 1997) by Berry and Linoff. or reworked from one of the two sources below 1.CRM Sources The vast majority of these CRM slides has been borrowed. Customer Relationship Management Through Data Mining. Michael Berry and Gordon Linoff. SAS Institute. 2000 Michael Berry and Gordon Linoff. 2000 2. Mastering Data Mining. John Wiley & Sons.

Decision Trees and Churn Modeling Learn about decision trees what are they? how are they built? advantages versus disadvantages Case study from the cellular telephone industry problem definition looking at the results other techniques for modeling churn 213 .

leaves. and splits 214 . nodes.Data Mining with Decision Trees A decision tree is a set of rules represented in a tree structure Easy to understand how predictions are made Easy to build and visualize Can be used for binary or multiple outcomes Roots.

Data Mining with Decision Trees--continued We build the decision tree using the training and test sets The decision tree allows us to make predictions understand why certain predictions make sense understand which variables are most important spot unexpected patterns There is one path from the root to any leaf A given data record is associated with a single leaf 215 .

.A Path Through the Decision Tree At each node.g. a decision is made--which variable to split These variables are the most important Each leaf should be as pure as possible (e. nearly all churns) All records landing at the same leaf get the same prediction A small example follows 216 .

then not a widget buyer Rule 2.A Decision Tree for Widget Buyers Res  NY 1 yes 6 no No Age > 35 2 yes 3 no 11 yes 9 no No Res = NY 10 yes 3 no Age <= 35 8 yes Rule 1. If residence not NY. then not a widget buyer Accuracy = Yes # correctly classified total # 638  0. 1997) = . then a widget buyer Adapted from (Dhar & Stein. If residence NY and age <= 35. If residence NY and age > 35.85 20 217 Rule 3.

all churners on left and non-churners on right For each child of the root node. we seek to maximize purity Eventually. we again search for the best split i. the process stops no good split available or leaves are pure 218 ..Building a Decision Tree We start at the root node with all records in the training set Consider every split on every variable Choose the one that maximizes a measure of purity ideally.e.

we prune the tree using the test set to prune is to simplify After the tree has been built.Building a Decision Tree--continued The above process is sometimes called recursive partitioning To avoid overfitting. each leaf node has a score A leaf score may be the likelihood that the more common class arises in training and test sets 219 .

965 No Size 96.112 records (including Sam’s) are associated with this leaf 96.5% 11.112 220 .5% Sam gets a score of .The Leaves Contain the Scores Overall. the most common class is No 11.5% are not fraudulent Yes 3.

428 0.957 0102 Will 0112 Pat 0116 David 0117 Frank 0118 Ethel This example shows the relationship between records in a table and decision tree leaves Remember.283 0.892 0.Scoring Using a Decision Tree ID 1 2 3 4 5 Name State MA WY ID MS NE Score 0.957 0. each record will fall in exactly one leaf 221 .

5%) The tree will be built using training set data only222 .814 Churn data from the cellular industry Begin at the root node The training set is twice as large as the test set Both have approximately the same percentage of churners (13.8% 86.5% 39.5% 86.628 13.A Real Life Example: Predicting Churn Training Test 13.2% 19.

0% 96.8% 3.529 28.678 14.607 223 .9% 85.7% 71. test set on right  3.3% 70.361 15.0% 11.5% 86.7% 13.8% Handset Churn Rate < 3.8% 86.812 Training set on left.4% 11.5% 39.112 5.628 < 0.5% 97.7% 2.5% 3.3% 5.6% 84.155 29.The First Split 13.2% 19.1% 23.

12 224 .8 = 2. we see that the churn rate has more than doubled Far-right child has a lift of 29. and high churn rates If we look at the child on the far right.The First Split--continued The first split is made on the variable “Handset Churn Rate” Handsets drive churn in the cellular industry So. first split is no surprise The algorithm splits the root node into three groups low.3/13. medium.

0% 11.678 Handset Churn Rate < 3.0056 15.7% 2.361 < 0.3% 32.6% 57  3.7% 110 66.8% 28.5% 86.3% 5.112 5.0% 440 27.6% 42.1% 55 < 88455 52.0% 48.0% 34.814 < 0.8% 14.9% 85.8% 86.0% 25 >= 88455 25.2% 60.5% 3.6% 84.1% 54 44.18 27.9% 29.18 39.2% 19.628 13.As the Tree Grows Bigger.3% 70.8% 148 40.5% 39.155 29.4% 59.7% 71.9% 74.0% 47 70.5% 97.1% 218 CALL0/CALL3 58.4% 219 99 Total Amt Overdue < 4855 67.1% 23.0% 56.4% 11.9% 72. Nodes and Leaves Are Added 13.0% 96.4% 55.0% 73.529 < 0.607 >= 0.7% 3.6% 27 225 .0% 43.

the voluntary churn rate goes down It is easy to turn the decision tree into rules 226 .As the Tree Grows--continued CALL0 = calls in the most recent month CALL3 = calls four months ago CALL0/ CALL3 is a derived variable The number of calls has been decreasing Total Amt Overdue = total amount of money overdue on account When a large amount of money is due.

0% (on test set) If Handset Churn Rate  3.Three of the Best Rules If Handset Churn Rate  3.0% (on test set) If Handset Churn Rate  3.0056 AND Total Amt Overdue < 4855 THEN churn likelihood is 66.8% AND CALL0/CALL3 < 0.8% AND CALL0/CALL3 < 0.18 THEN churn likelihood is 40.0056 AND Total Amt Overdue < 88455 THEN churn likelihood is 52.4% (on test set) 227 .8% AND CALL0/CALL3 < 0.

From Trees to Boxes Another useful visualization tool is a box chart The lines in the box chart represent specific rules The size of the box corresponds to the amount of data at a leaf The darkness of a box can indicate the number of churners at a leaf (not used here) We use HCR and C03 as abbreviations for Handset Churn Rate and CALL0/CALL3 Fill in three boxes in top right on page 228 228 .

7%  HCR < 3.18 HCR C03 229 .From Trees to Boxes--continued HCR HCR < 0.8% C03 0.0056 3.8% 0.8% 0.18% < 0.7% C03 3.

but it is not our focus here 230 . they have two things in common splits are preferred where the children are similar in size splits are preferred where each child is as pure as possible Most algorithms seek to maximize the purity of each of the children This can be expressed mathematically.Finding a Good Split at a Decision Tree Node There are many ways to find a good split But.

Synonymous Splits There are many ways to determine that the number of calls is decreasing low amount paid for calls small number of local calls small amount paid for international calls Different variables may be highly correlated or essentially synonymous Decision trees choose one of a set of synonymous variables to split the data These variables may become input variables to neural network models 231 .

Better Ones 232 .Lurking Inside Big Complex Decision Trees Are Simpler.

Problem: Overfitting the Data 13.7% 2.3% 70.814  Good! The training set and test set are about the same Handset Churn Rate 3.7% 71.8% 86.5% 86.607  Good! The training set and test set are about the same  Good! The training set and test set are about the same CALL0/CALL3 < 0.155 29.0% 42.0% 219 Total 70.0% 25 X Ouch! The tree has memorized the training set.5% 39.6% 43.1% 55 56.628 13.8%  28.9% 29.2% 19.0056 58.4% 99 Amt Overdue < 88455 52. but not generalized 233 .0% 48.3% 5.

70 0.73 0.72 0.71 0.74 0.69 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 TRAIN TEST Number of Leaves 234 .Sometimes Overfitting is Evident by Looking at the Accuracy of the Tree Proportion Correctly Classified 0.75 0.76 0.

Pruning the Tree Prune here This is a good model Error rate is minimized on unseen data Error Rate Number of Leaves 235 .

Connecting Decision Trees and Lift A decision tree relating to fraud within the insurance industry is presented next Assume the analysis is based on test set data The cumulative gains chart consists of a series of line segments Each segment corresponds to a leaf of the tree The slope of each line corresponds to the lift at that leaf The length of each segment corresponds to the number of records at the leaf 236 .

0% 100.8% Yes (370) 40.0% 90.0% Rural (122) No (56) 45.2% Yes (436) 41.2% Yes (36) 2.0% 0.5% Yes (435) 23.0% 80.4% Yes (452) 30.0% 30.0% 80.8% Third Party(439) No (423) 96.0% 10.4% Yes (16) 3.0% 10.0% Base Policy All Perils (1481) Collision (1840) No (1029) 69.0% 70.6% Fault Policy Holder (1042) No (606) 58.0% 60.5% No (1405) 76.0% 50.0% Yes (923) 20.6% 100.0% 0.0% 30.0% 20.0% 20.Relationship Between a Cumulative Gains Chart and a Decision Tree No (3692) 80.0% 50.0% 40.0% 40.9% Yes (66) 54.0% 90.1% Urban (920) No (550) 59.0% 60.2% Decision Tree Baseline Adapted from Berry & Linoff (2000) % Claims 237 .0% 70.8% Accident Area %Fraud Liability (1294) No (1258) 97.

Decision Trees: A Summary It is easy to understand results Can be applied to categorical and ordered inputs Finds inputs (factors) that have the biggest impact on the output Works best for binary outputs Powerful software is available Care must be taken to avoid overfitting 238 .

and expected attrition 239 . forced. it becomes cheaper to retain than acquire customers Review the difference between voluntary.Case Study: Churn Prediction Predicting who is likely to churn at a cellular phone company The cellular telephone industry is rapidly maturing The cost of acquiring new customers is rather high A handset is discounted in exchange for a time commitment As the market matures.

S.The Problem This case study took place in a foreign country Outside of North America and Europe Mobil telephones are rather advanced in newly industrialized countries – used instead of fixed line technology The penetration of mobile phones in the U. was 25% in 1999 240 .

continued The penetration of mobile phones in Finland was >50% in 1999 The company is the dominant cellular carrier in its market It has 5 million customers Four or five competitors each have about 2 million customers 241 .The Problem-.

The Problem-. using consultants The work took place over a two month period 242 .continued The company has a churn rate of about 1% per month It expects its churn rate to jump There are companies in this industry with churn rates of 4% per month The company wants to assign a propensity-to-churn score to each customer and put the resulting model into production ASAP They decided to develop the expertise in-house.

9% 1.000.000 5.500.000 3.1% Elite customers exceeded some threshold of spending in the previous year They have been around at least one year Why is their churn rate higher? 243 .3% 0.500.Customer Base Segment Elite Non Elite Total # Customers 1.000 % of Customers 30 70 100 Churn Rate 1.

etc. zip code. Handset information – type. service activation date.Data Inputs Customer information file – telephone number. additional services. Elite Club grade. Dealer information – marketing region and size Billing history Historical churn rate information by demographics and handset type 244 . etc. billing plan. weight. manufacturer.

August and September churners are used We want to use the model to make predictions for November 245 .The Model Set Uses Overlapping Time Windows Jan Feb Mar Apr May Model Set Model Set Score Set 6 5 6 4 5 3 4 6 2 3 5 Jun 1 2 4 1 3 2 Jul Aug +1 +1 1 +1 Sep Oct Nov For the model set.

20th.000 Elite customers who are most likely to churn in October The revised problem invites action 246 . provide a list of the 10.Solving the Right Business Problem Initial problem: assign a churn score to all customers Complicating issues new customers have little call history telephones? individuals? families? voluntary churn versus involuntary churn how will the results be used? Revised problem: by Sept.

Build a Number of Churn Models Numerous decision tree models were built It was discovered that some customers were taking advantage of the family plan to obtain better handsets at minimal cost These customers used the marketing campaigns to their own advantage Different parameters were used to build different models The density of churners in the model set was one important parameter (50% worked well) 247 .

mass-produced goods 248 .Database Marketing and the Catalog Industry Catalogs are everywhere For many people. the reaction is the same Catalogs are very old the first Sears catalog was published in 1888 Catalogs flourished over the next half century railroad. catalogs are the preferred method of shopping Their resemblance to magazines is intentional For many people. coast-to-coast mail service.

.Catalogs are Part of the Retail Business Catalog retailers know their customers as well as their merchandise They communicate one-to-one and not just through branding and mass market advertising Ability to customize catalogs Closely related to B2C Web retailing Many companies (e. catalogs.g. and a retail Web site 249 . Eddie Bauer) have stores.

The Catalog Industry $95 billion industry in U.S. (Catalog Age. retail sales B2C Web retail sales are approximately 1% of U. 1998) Catalog sales are approximately 10% of all U.S. retail sales (1999) Most catalogs now have an online presence It is becoming difficult to distinguish between Web retailers and catalog companies 250 .S.

What do Catalogers Care About? Merchandising how to display and promote the sale of goods and services? Layout where to place items in the catalog? which models to use? Response closer to the subject of this course personalization. customization. recommendation which of several pre-assembled catalogs should be sent to a specific customer based on an estimate of his response to each? 251 .

Example of a Big Catalog: Fingerhut Manny Fingerhut started database marketing in 1948 in Minneapolis sold seat covers on credit to new car buyers Fingerhut and specialty catalogs emerged over time As of the late 1990s $2 billion in annual sales 400 million catalogs/year (> 1 million/day) terabytes of transaction history customer and prospect file with 1.400 fields describing 30 million households 252 .

and Web are equal partners Many are aimed at a very narrow audience 253 . retail stores.95” is a way to hide higher interest rates the most profitable customers don’t have to be the wealthiest Victoria’s Secret sees its catalog as part of its branding every mailbox should have one Eddie Bauer has a multi-channel strategy catalog.Different Catalogers Have Different Approaches Fingerhut aims for a less affluent audience “4 easy payments of $19.

The Luxury of Pooled Data Unlike other industries. L. you may have observed that a purchase from catalog A sometimes triggers the arrival of catalog B 254 .L. Bean) As a consumer.g.S. Eddie Bauer) can decide to send you a catalog because of a purchase you made from a competing catalog (e.. the catalog industry promotes the sharing of data through a company called Abacus U. Air doesn’t know about my travels on TWA or Northwest Airlines But a catalog company (e..g.

100 member companies maintains a database of over 2 billion transactions includes the vast majority of U. catalog purchases sells summarized data to member companies details of individual transactions are not revealed facilitates industry-wide modeling DoubleClick a leader in Web advertising.S.Abacus and the World Wide Web Abacus is the catalog industry infomediary 1. now owns Abacus indicates the convergence of online and off-line retailing 255 .

household Acxiom.) database marketers use this data to assemble mailing lists Available data internal data on customers industry-wide data from Abacus demographic data from household data vendors 256 .S. Experian. R.Another Source of Data Household data vendors companies that compile data on every U. Equifax. Claritas. First Data. etc.L. lifestyle. Polk hundreds of variables (demographic.

What Some Big Catalog Companies are Doing with Data Mining Neural networks are being used to forecast call center staffing needs New catalog creation what should go into it? which products should be grouped together? who should receive the catalog? Campaign optimization 257 .

Eddie Bauer Eddie Bauer is an interesting case They seek to maintain a unified view of the customer across three channels give the customer a unified view of Eddie Bauer They are integrating all channels for CRM 400 stores catalogs Web sales 258 .

Vermont Country Store (VCS) Eddie Bauer and other large companies use CRM extensively What about smaller companies? VCS is a successful family-owned business VCS is a $60 million catalog retailer (late 1990s) VCS first used CRM to improve the targeting of their catalog to increase the response rate They used SAS Enterprise Miner to achieve a dramatic return on investment 259 .

Orton in 1945. he had helped his father and grandfather run a Vermont country store His first catalog went out in 1945 or 1946 36 items 12 pages mailed to 1000 friends and acquaintances contained articles and editorials had the feel of a magazine 260 .Early Company History Founded by V. after he returned from WWII As a child.

as a whole.The Next Generation Son Lyman Orton took over in the 1970s he remains as owner and chairman the CEO is not a family member Lyman focused on the catalog side of the business VCS grew by about 50% per year during the 1980s without data mining The catalog industry. expanded rapidly during the 1980s 261 .

The Industry Matures As the catalog industry prospered. more than one-third of catalog firms were operating in the red VCS was still profitable. it attracted new entrants Catalogs began to fill every mailbox Response rates declined Growth rates declined The cost of paper and postage increased rapidly By 1995. but concerned 262 .

to people who do” $60 million in annual sales 350 employees Use Enterprise Miner to build response (predictive) models VCS sells the notion of a simple and wholesome (old-fashioned) New England lifestyle 263 .Vermont Country Store Today They seek to “sell merchandise that doesn’t come back -.

Business Problem: Find the Right Customers The VCS vision does not appeal to everyone Historical data on responders and non-responders is used to predict who will respond to the next catalog Focus on existing customers upside: we need to select a subset of a pre-determined set of customers downside: we are unable to identify outstanding new prospects The goal of response modeling is to find the right customers 264 .

Possible Objectives or Goals Increase response rate Increase revenue Decrease mailing costs Increase profit Increase reactivation rate for dormant customers Increase order values Decrease product returns 265 .

RFM: Common Sense in Retailing Recency: customers who have recently made a purchase are likely to purchase again Frequency: customers who make frequent purchases are likely to purchase again Monetary: customers who have spent a lot of money in the past are likely to spend more money now Each one of these variables is positively correlated with response to an offer RFM combines all three variables 266 .

Where to Begin Any of the previous goals can be addressed via data mining The first step is to define the goal precisely The specific goal determines the target variable Build the customer signatures orders placed by customer over time items purchased by category over time customer information from internal sources use customer zip code to add external data purchase household and neighborhood level data 267 .

000 268 .115.591.593.000 9.000 5.S.134.263.965.925.FEMALE Under 5 years 5 to 9 years 10 to 14 years 15 to 19 years 20 to 24 years 25 to 29 years 30 to 34 years 35 to 39 years 40 to 44 years 45 to 49 years 50 to 54 years 9.000 11.137.095.000 9.201.085.000 1.000 MALE 9.000 8.000 9.000 3.000 6.872.000 5.906. we compare RFM and more sophisticated approaches 55 to 59 years 60 to 64 years 65 to 69 years 70 to 74 years 75 to 79 years 80 to 84 years 85 to 89 years 90 to 94 years 95 to 99 years 100 years & over 6.025.231.000 276.616.000 4.277.715.000 10.000 51.993.000 9.021.663. population (as of Nov.000 9.000 4.000 8.202.000 9.000 10.070.000 11.573.800.341.836.644.000 11.190.000 1.000 RFM and Beyond RFM uses recency/frequency/ monetary cells to determine who receives a catalog On left is the U.000 3. 1999) by age and gender In this section.000 8.300.000 313.000 10.000 4.000 864.000 11.000 9.246.000 9.321.723.000 9.000 9.000 4.900.800.000 79.000 851.000 3.000 11.

90% 48.74% 49.09% 55.89% 22.80% 32.70% 49.59% 49.23% 62.75% 82.49% 50.26% 50.22% 51.41% 50.30% 50.74% 54.25% 17.74% 75 – 79 year old men fought in WWII 269 .64% 48.98% 52.57% 73.43% 26.51% 49.17% 51.77% 37.36% 51.20% 67.11% 77.91% 44.07% MALE 51.14% 41.89% 51.26% 45.86% 58.02% 47.83% 48.40% 50.93% The Cells of the Table Below are Interesting More boys than girls until age 25 More females than males after age 25 25 to 29 years 30 to 34 years 35 to 39 years 40 to 44 years 45 to 49 years 50 to 54 years 55 to 59 years 60 to 64 years 65 to 69 years 70 to 74 years 75 to 79 years 80 to 84 years 85 to 89 years 90 to 94 years 95 to 99 years 100 years & over 50.78% 48.10% 51.11% 48.60% 49.FEMALE Under 5 years 5 to 9 years 10 to 14 years 15 to 19 years 20 to 24 years 48.26% 49.

.g. CapitalOne) The insurance industry uses cell-based approaches to assess risk Market research also uses cell-based approaches. especially when the cells reflect demographics that can be used to predict purchasing behaviors 270 .Cell-Based Approaches are Very Popular RFM is used often in the catalog industry Cell-based approaches are used to assess credit risk in the banking industry (e.

. pricing. bushiness of mustache Measure item of interest (e.g.RFM is a Powerful General Methodology Divide the “population” into cells based on known variables e.g.. focus on the cells of interest through selection. age. via a test mailing list) In full-sized mailing. sex. response) for each cell (e.g. income.. advertising 271 .

but important for comparison purposes 272 . when the catalog industry was growing rapidly Proved to be very valuable. so it spread rapidly Became less effective in the 1990s at a time when costs (including paper costs) started rising rapidly Not a customer-centric approach.RFM: The Baseline Method for the Catalog Industry RFM was developed in the 1980s.

Sort List by Recency Sort customers from most recent purchase date to least recent arrangement is based entirely on rank Break the list into equal-sized sections called quantiles top. medium. bottom high. low quartiles quintiles deciles This quantile becomes the R component of the RFM cell assignment 273 .

break the list into quantiles But how do we measure frequency? total number of orders divided by the number of months since the first order average number of orders per month over the past year The quantile becomes the F component of the RFM cell assignment 274 .Sort List by Frequency Resort the list by frequency.

Sort List by Monetary Value Resort the list by monetary value. it tends to be big 275 . break the list into quantiles But. imagine a customer who has not placed an order in a little while doesn’t purchase that often when she does place an order. what is the right measure? total lifetime spending average dollars per order The quantile becomes the M component of the RFM cell assignment Now.

From Sorted List to RFM Buckets 341 1 2 3 4 5 1 2 3 4 5 1 2 3 Recency 4 5 Frequency Monetary 276 .

RFM Cube Recency = 5 Recency = 4 Recency = 3 Recency = 2 511 521 531 541 551 411 311 211 111 Frequency = 1 421 321 221 121 Frequency = 2 431 331 231 131 Frequency = 3 441 341 241 141 Frequency = 4 451 351 251 151 Frequency = 5 Recency = 1 277 .

0% 20.0% 20.0% 5.0% 10.0% 15.0% 10.0% 5.RFM Cells are Not All the Same Size 211 221 212 222 Consider the small example on left 111 121 112 122 Do the numbers make sense? 15.0% 278 .

0% 15.33% 0.62% The better quantiles along each axis generally have higher response As expected.0% 20.38% 2.0% 5.00% 1.0% 5.Each Cell Has a Different Response Rate RFM CELL PROPORTION RESPONSE 111 112 121 122 211 212 221 222 10.09% 2.70% 0.99% 1.0% 3.0% 20.71% 2.0% 10.0% 15. the 111 cell has the best response and cell 222 has the worst 279 .

Using RFM Cells to Determine Which Customers Should Receive a Catalog Given a customer list of one million Build a cube of 5 x 5 x 5 = 125 RFM cells Test mail to a random sample with all cells represented Given a budget that will cover test mail plus sending out 100.000 catalogs Mail catalogs to customers in top cells with respect to expected response to reach 100.000 280 .

the way we do in building a predictive model 281 . it does not use a test set concept.RFM is a Testing Methodology RFM involves testing the marketing environment in order to find the best cells direct test historical data Such testing makes good marketing sense However.

00 revenue per response = $40.00 let r = response rate profit = 39r – (1 – r) profit > 0  39r > 1 – r 40r > 1  r > 2.5% response in test mailing 282 .00 net revenue per response = $39.Choosing Profitable RFM Cells Calculate break-even response rate Example cost per piece = $1.5% mail to any cell with > 2.

99% 1.0% 3.09% 2.0% 5.Choosing Profitable Cells RFM CELL PROPORTION RESPONSE If we need a 2.71% 2.00% 1.33% 0.70% 0.0% 20.0% 10. we choose cells 111 and 121 111 112 121 122 211 212 221 222 10.62% These account for 25% of the population 283 .0% 20.0% 5.38% 2.0% 15.5% response rate to break even.0% 15.

Using RFM Cells as Control Groups VCS wanted to test a new catalog designed to appeal to 53-71 year olds The motivation was to exploit nostalgia for teenage and young adult years The new catalog included color images of products black and white images of 1950s and 1960s The test mailing compares target segment against others from the same RFM cell 284 .

we seek to gain a basic understanding of neural network models What are the pitfalls to be avoided? In this section.Using Neural Networks for Response Modeling RFM is our baseline method RFM has a number of drawbacks At this time. we discuss the neural network response model built by VCS 285 .

Problems with the RFM Approach RFM cell assignments are not scores is cell 155 better than cell 311? a test mailing is required RFM approach misses important segments Christmas shoppers: in November they may look like they have stopped buying from catalog there may be pockets of good responders in “bad” RFM cells there may be pockets of bad responders in “good” RFM cells 286 .

Problems with the RFM Approach -continued Proliferation of cells there can be a large number of RFM cells RFM approach hits the same customers over and over the same people end up in the “best” RFM cells these customers suffer from “offer fatigue” RFM approach does not make use of some valuable data VCS knows where customers live (demographics) VCS knows about past behavior (purchases. categories) VCS knows which customers are gift givers 287 . returns.

this doesn’t always make sense Predictive models are more powerful than profiles cell definitions are profiles profiling assigns a customer to a cell whose aggregate behavior is measured predictive models use input variables to predict future behavior of each customer 288 .Problems with RFM Approach -.continued Focus on cells rather than individuals cell-based marketing treats everyone who falls into the same basket the same way because cell definitions tend to be simplistic.

neural networks. and decision trees 289 . VCS became less enchanted with the technique In 1998. they were ready to try something new VCS worked with SAS consultants to see if Enterprise Miner could achieve better results They compared RFM. regression.Back to VCS After some initial success with RFM in the 1980s.

neural network. and decision tree models to predict who would respond to two other mailings that were also in the past. and profit assuming models had been used 290 . but more recent than the model set data October mailing November mailing Calculate expected response.Experimental Design The model set data came from previous catalog mailings with data on responders and nonresponders Using this data. VCS built RFM. revenues. regression.

& M are each represented 291 . F. or both Number of catalogs purchased Number of days since last order Number of purchases from the various catalog types Dollars spent per quarter going back several years Note that R. credit card.Available Data on Each Customer Household ID Number of quarters with at least one order placed Flags indicating payment by personal check.

neural networks.How Models were Judged Compare regression. and decision trees with RFM baseline with respect to percent increase in dollar sales Look at return on investment in data mining project How do we guess sales since the comparison is hypothetical? SAS and VCS were very clever 292 .

use average spending of recipients with the same score 293 . given that we didn’t? Each model was used to score the list and choose its own top 1. use actual dollars spent for non-recipients.4 million prospects for actual recipients.Using Historic Response Data to Compare the Models Challenger models will pick a different mailing list than the one generated using the RFM approach How do we guess what someone would have spent had we sent her a catalog.

86% to 12. depending on the specific catalog The model yielded a return on investment of 1.83% perhaps.The Results The neural network model was the model selected by VCS The model predicted an increase in sales ranging from 2.182% More detailed information is confidential Let’s revisit the topic of neural networks 294 .

Neural Network Architecture output layer hidden layer input layer inputs are usually columns from a database 295 .

weighted inputs are summed and passed through a transfer function i3 w1 w2 w3 i1 i2 inputs The output value is between 0 and 1 The output node functions similarly 296 .Inside Each Hidden Node output Inputs are real numbers between -1 and 1 (data is transformed) Links have weights At each node.

Training a Neural Network Given a pre-specified training set and random initial link weights observe one pattern (a single inputs/output combination) at a time iteratively adjust the link weights stop when final weights “best” represent the general relationship between inputs/output combinations This is called backpropagation An alternative approach is to use genetic algorithms 297 .

testing. and validation sets Train the neural network using the training set until the error is minimized or the link weights no longer change Use the test set to choose the best set of weights for the model Compare models and predict performance using 298 the validation set .Procedure for Using Neural Networks Transform all the input variables to be between -1 and +1 and output variables between 0 and 1 Divide the data into training.

re-renter of Ryder trucks in the training set.Neural Networks: Pitfalls to be Avoided Decision trees can easily handle hundreds of input variables Neural networks cannot more input nodes  more link weights to be adjusted  the larger the training set  computationally burdensome one solution: build a decision tree first and use the variables that appear high in the decision tree in the neural network Categorical variables e. define a variable that takes on the value 1 if the customer is a re-renter and 0 otherwise 299 ..g.

and the homeless would be obscured possible solutions: throw out outliers or perform transformations using logarithms or square roots 300 . college professors. Bill Gates is assigned 1 and everyone else is assigned 0 differences between cardiologists.Neural Networks: Pitfalls to be Avoided-Continued Be on the lookout for data skew caused by large outliers e. in looking at net worth..g.

2000) Data visualization: application to college selection CRM recap 301 .Final Thoughts on CRM Cross-sell models: see Chapter 10 in Berry and Linoff (Wiley.

John Wiley & Sons. in preparing these slides 302 .CRM Sources The vast majority of these CRM slides has been borrowed. SAS Institute. Michael Berry and Gordon Linoff. Mastering Data Mining. Michael Berry and Gordon Linoff. Customer Relationship Management Through Data Mining. 2000 2. 2000 I have also consulted Data Mining Techniques (Wiley. or reworked from one of the two sources below: 1. 1997) by Berry and Linoff. adapted.

9082320/ 43:38003/. 7747 #.!7:3390%700 !7:30070 %88.44/24/0 77477.90 :2-07410..08  .9.

1  ..3.:2:.47708543/894.30.9 0.80/439089809/.843%7008.0.3/19 /0.3.9.47708543/89490 3:2-074170.939417.930..4330.0 3/:897857080390/309 88:2090.802039.1 %0039410.:/939038:7.38.9900.99.0.43889841.141909700 %08450410.47708543/8949019.8070841 308020398 .. %0.79.802039.47/8.9.888-.843970070..

79  4    08            #:7.843%700 4    08    .0.3/.9438509003.  4    08    &7-.590/1742077 3411  . : / 4843  4     08    .  7 .9.4/07   4     08    .80!4.38 .79.28  ..8030 /.:2:.:9 !4.#0.0.843%700 . !078  4     08    ./03970.3  4    08                                      0.-9  4    08    %7/!.

74:95:98 !4071:8419.0394.550/94.7 980.9047.702:89-09..071993  ....3/708:98 .843%7008$:22.4/4..3-0..9 43904:95:9 478-089147-3.-0 .9478 9.708..9...090-08925..3/47/070/35:98 3/835:98 1.894:3/0789.0.

7 5430.30147.4:39.907 .4:390/30.39.42292039 8902.425.6:7330.80$9:/:73!70/.9.790054303/:89787.943 !70/...0.920 .9348094.090/110703..0:.9:708 9-0.0:.:894207887.5/2.7 147.7092.:8942078 #0..0-09003.4208.:73..3.6:70.0/  .997943  .3 %0.9:73 %0.50794 709.90/.3/8098/8.48941.3/050.

%0!74-02 %8.03....907.707.4:39708 :80/3890././4110/ 3090.3..9434124-054308390& $ .3/:7450 4-90054308.14703.0/..0/330 3/:897.8089:/9445.4:397 :98/041479207.344 %0503097.8 3  .

0:..3/.:8942078  ...8243.425.:8942078 4:7471.9434124-05430833.4250994780.8  3 %0.4393:0/ %0503097..0.7.39.3890/423.709 9.-4:9 243.%0!74-02 .77073 982.0.

%0!74-02 .07.:737.9094:25 %070.:737.9898.-4:95072439 9050.:894207.425.3.:73 8.438:9.9424395074/  .0459005079803 4:80 :83 .470940..:737.39894.943$! %0/0.57450389 94 .3.9041./0/94/0.3083983/:8979.908 415072439 %0.398 %0479445..3/5:990708:9324/0 394574/:.8.70.425...883.425.04.4393:0/ %0.

:89420780.7 8907.80 $02039 90 4390 %49.00/0/842097084/41 8503/3390570.9007  .90      90.7 %0.0-003..4:80.:894207.894300.90. :8942078      41:8942078    :73#.:737.74:3/.

9.3/80931472.0.08 90 :-7.3/.:737. 0.8 .70937043..9:707 09  09...3 5..9.943 2.943/.943-/0247../0 807..3:1. ..3/809950  .4/0 .0731472.//943.943 950 2.90 09.5.94310 90054303:2-07  -35.9031472.35:98 :89420731472.3/80 38947 8947..807.

3 0. 479024/0809 ::89.%04/0$09&808 .0570/.470$09            :3       : :     $05 .553 %203/48 .:73078 ..9 4. 4/0$09 4/0$09 $.3994:809024/0942.07.02-07  .9438147 4.7 57 .3/$05902-07.70:80/ 0.

4:39.:89420784.4:39.:73 490708:98-0:80/ #0.47094../:.908.:73 3 .94-07 %070.7.208 .574-02.:8942078 425.:8942078.894190  90..80/574-023..078:83.943  .0990.390#9:83088!74-02 39.8947 900543083/.702489094.9388:08 30.81.:73.:738.883./0..7.80/574-02-$059  9 574.$4...

25.8430 25479.39.7.843970024/08070-:9 9..9232.20907  470/0  .39.7093.8/8.:7307839024/0809.070/9..:2-0741:734/08 :2074:8/0.3944-9./../.:8942078:80/902.395.0 11070395.25..041901.98420.489 %080.3-09907 .4.3 .:/.3/8098.209078070:80/94-://1107039 24/08 %0/038941.3894 90743.:89420780709.7.

9438908.4..039:7 7.30883903943.9.0/ 44/8  .89 94 .1.3/90.9.9.4.88 574/:. 472.0790309..9.4814:780/4..700.074/ 901789$0.48.0 2.20 ..78.9.48./ .4 3/:897 .709057010770/ 2094/4184553 %0770802-.80.892.350450 9070.807.48.3.9..7093.0942.85:-80/3 .9.74.07070 472.-..4.350450 .70.

.9.794190#09.709 .8907207.0894708  .308 0  //0.:83088 .48..9.90430 94 430 .709.3/2.422:3..3/.0-890  .:8942078 .07983 -994.3/349:89974:-7.3/80 %0.:07 ..425.80.882.48 .07834907.3.9.:89420../.3 .90/940-709.9.3/3.48 48070..4709.70!.

08.55742.48.0.078..:994/893:8-09003 0-709.8..9.8.4834.9.9.08 0-709.34330570803.0 98-0.9.70.423/11.3/.425.08  489.43/:897 -433/:8973& $  .40  .8.9041 & $ 709.08.9.& $  709.90 41.4.70.55742..%0.308  .

..4884:/-0 803994.  .4078.1.8802-0/.570 .9..4 ..9/4.09028390..9.4:9 070945.4180..3/5742490908.850.943 .9..480794908:-0.943 70...07.90418 70854380940.42203/.80/43.70-4:9 07..30892.04144/8.24/0894:80 #0854380 .943 .3/ 807.:8942.:894207-.3/83 494/85.08 .94198.4:780 507843.

802.4307:9 ..9..548 84/80.4802070/4.25041.-.90 8 -433.33:..07843...9.3/850.08  243.33307:989..8.7-:078 307:9.790//.48.9.9.07920 84190.9.70/99430.70933 3330.9.4.

7 243.0.

9109  10/8/08.:894207.7-3  2434:804/8  .38. 907./.9438947 .-9084197..3/574850.

.5.4078.85.9.947.4.11:039.20/..2039841  8.073.0 0.793078 .3.70.2:9 .90 .-0.9.:07.85..0  .794198-7.94/0073907089 7.:/03.908 90248957419.088.894708 .774.9..:8942078/43 9.01107039 5574..3/0-..709800898..094-0900.9..0430 //0..7006:.8.28147.-484:/.3/3 0.08 307:9. 8$0.1107039.330897.:/03.9089 '.072.4 709.

.974:.77. 5:7..43/:897 57424908908..9.804:2.3 0  //0..425093..9.425.:8 & $ 7/4083 934..9. 5:7.4  .04-807.4842092089707890 .9./094803/4:..4-0./01742.801742..7341/.438:207 4:2.:8041.:07 .4 0    0.%0::741!440/..0843% 474790897308 :9..3 8..-4:9297.41.9.425..9.9.4.0/9....0/-...9.3 .3 /0.9..... &3049073/:89708 90.

.9083/:897 /024/03 4:-0.94202-07.9.808 8088:22.9438.47941& $ .425.07-4397.:/0890.:8890.0.3/411 30 709.7   202-07.39..70//.43/:89731420/.0414330.0/ 1.7034970.308 2.0.-..45:7.38.38...3  .97..80414.90890.07983 34438-../:.9.892..38..425.3/9047//00-.9../.9438 3. .9./0730-.43.:8 3/./.:8.308 /09.0703.9.8413/.-..

9.9.8 507..17424:804//.43.8802-02.9.79.-08 /0247.-0/.5.42 789. ./.03/478  ..709078:8098/. /.9.9.1742-.03/478 . 6:1.94.7.3 #  !4 :3/70/841.5..802..9.. 4:804//.9.9.3089.:8 /0247.9.041./.9. 39073.430.07& $ 4:804/ ..9.4250/.-. 10890 09.425.9.3898 .34907$4:7.:8942078 3/:897 /0/.

9884:/-074:50/940907 484:/70.89.984:/43949 .9.. .9.943 .309478..943  .113300/8 0.25..0390789.4.34592.574/:.70.33 0:7.4 .9.4425.9.308.9$420..70-03:80/941470.70 439.0.090..

.48 0-8.39..703907.090.3308147#  894708 .9..08  ..:310/.041//0.//0.:894207.7488 9700.:894207.:07 %0.:310/.:07 //0.93..80 %080094 2..04190.339070893.3.:078.3308 .

0881:1.90 8 '$1789:80/#942574. /7.425..70.4709.8:.9.07.0892039  .3/4907.308:80# 09038.4943.0909.:07.2 430/-:83088 '$8.425.9.90 %0:80/$$3907578030794.0 .07 ..70.9.9.709:73433.0.2.0..709341 907.-4:982.308 '$8.8090708543807. 243.'0724394:397$9470 '$ //0..

439..30  ./050/81.9.19070709:730/ 1742 8./ 0..4:39789470 81789.6:./9010041..'072439.3.08 2.3/1..79.907 7:3.3/..8 .30/.7425.3/7.08 .0/94 1703/8.38947 4:3/0/-'  79433 .907.40394:9347 9028 5.39.3/0/947.2.08.

48/04190-:83088 '$70-.3/0/7.:80/4390..-4:9 5070.84307.72.43/:897 .233 %0.3 79439444.2202-07 2.5/ /:7390 8  .40 05.943 $432..314.3 90 8349.8..7/:7390 8 94:9/.07390 8 0702.9.1.9.%0090307.38.3/.9.

908/0.-4 #08543807.5/  24709.072.3430 97/41.9.30/ 7497.489415.88957419.43/:89757485070/ 9.507.80/7.-0 -:9.908/0.41728070 4507.70.398 .48-0.90/30 0397.9:708 890...30/ %0.9.9.0730/  ..03.43.9339070/ '$.%03/:897.3/5489.997.39410.

3/10890  ..33:.8250.8. %080094 80207..9.420 -.'0724394:397$9470%4/.0  24/08 '$808903494341.9/4083 9.8430/ 03. 94504504/4  2433.3/809.3/408420 4/ 1.08  0254008 &803907578030794-:/70854380 570/.

-094/03914:989.:8430893..570 /0907230/809 41.43708543/078.4 4.:8942078 /438/00.07430 8947..417085438024/0389413/9079 .3/330 574850.940.98 %04.9.8:-80941.70:3.9.3/343 708543/0788 :80/94570/.:8942078 :58/00300/94800.843/408349.:83088!74-023/90#9 :8942078 %0'$./.94708543/9490309.9.:8942078  .550.

8070.39.90 3..:08 0.4898 3.8 3.!488-0 -0.70.8070..9.70.70.08474.80574/:.8047/07..70.70.90147/472.70.70.8057419 3.9709:738  .9.9437.3.80708543807.:8942078 3.03:0 0.802.

80..341107 #.9700..42-308.7..3 #0...7..700945:7.49412430 3905..0392.3 706:03..4770.808 .430419080.. 5:7.01706:0395:7.3 4309.-0885489.:89420784.03./0.90/ 97085438094.80.80.-08  .#42243$03803#09..:89420784..7.0.:894207842.070.700945:7..085039.700948503/2470243034 ..89.

..4/094.//09073...:8942074.80 %0850.07920 .9.4:84.07920 90285:7.0/-.0709403 34190570.4.3-0.08 :80.233 %017898905894/0130904..8.9.9:708 47/0785./09072308909.1.804:804/.0/.84:7..943174239073./.80/-.  .:89420731472.9.-0 :/90.:89420783.3/30-4744/0..//70880/.709.570.:8942075. /. 5:7.90474.7.

78 940.78  940. &3/070.78 940.78  0.78 940.78 940.78  940.03.78  940.78  940.78 940.78 940.78  940.78  940.78  940.78 940.78 940.78  4.78 940.07                                                                                  #.78 940.78  940.3/043/ #:80870..78  940.

1706:03..

425.7.  -.3/247084589.8414. 24309.90/ .08.4 3019890& $ 545:.5574..0..943 0.3/ 03/07 39880.0894/0907230 470..943 .70 #..9.0.08  .

78 940.78 940.78 940.78 940.74/203 14:93  .78  0.78  940.378 :39.78  940.-0 04.08.78  940. &3/070.78  940.78 4.0 470102.7039070893 470-489.78  940.78 940.78 940.78  940.3 2.78 940.78 940.07                                                                                              %0084190%.0  0.78  940.78 940.78  940.1907.78 940.089.78  940.

0 -.9/0247.5574.08  0850.9.88088.59.087010.9.5574.70:80/94.7097080.43/:897 0 -.95:7..3.70'07!45:.03/:897:808.3-0:80/94570/.83-0.80/.....478  .08.0894.80/5574..80/ .70/9 78390-..5574. 30 %038:7.08.0 -.8 9.333/:897 0  .5.7..80/.0390...8808878 .0 .7 #8:80/41903390.84:808.

07983  .943 57.420 -:83088412:89..0 0.8:70902413907089 0  70854380 1470.#8..80/43343.389 31: 80/2.7.0 80 3.08413907089 974:800./090 545:.094/44 ./.!4071:0307.90892.:84390.943 394.-08 0  .0 0  ..3 ..08 -.. .3 14.

..9.43/:897. -:925479.4898 3..0450/390 8 0390 ..507.43/:897 #..39 147.4898 89.:.8/0.5/ !74.87437.92003.790/ 7837.0390 8 .-0 8498570.200880110.8030094/14790 .:894207 .07.5/ 49./7.78435:754808  .5574.0/94-0.9.9.0397.#%0.5/ 0.:/35.425..9.

.3908 945 -49942  20/:2 4 6:. 80/80.390-0.0/6:.80/03970437.$47989-#0.7908 6:3908 /0. $479.9438.908939406:.3 70..90940.80/.3020398-.77.89 70.0.039 .425430394190# .8832039  .08 %86:.0395:7.420890#.:89420781742248970.03.

8832039  .8:701706:03.07905.420890.090178947/07 . -70. #084799089-1706:03.3908 :94/4020.890.3:2-074147/078/.390-0. 949.7 %06:.9089394 6:.425430394190 #..03:2-074147/07850724394.$47989 -706:03./0/-903:2-074124398 83.0.07.

30.0/.98907920.347/073.$47989-4309.809.:8942074 .420890..347/07 9903/894-0-  .07.7850747/07 %06:.3908 :9 .0.109208503/3 .:0 -70.8832039 4 2.425430394190 #.9089 3946:.0.390-0.7'..9900 /4083 95:7..7.:0 #084799089-24309.941903 0380/4085..83495...0/4.8:70 949.

098                 #0.742$4790/8994#:.7  . 4309.03. 706:03.

      706:03. #0.      706:03. #0.      706:03. #0.03.03.#:-0 #0.03.      706:03. #0.      706:03.  .03.03.

 0.#08.25043019 4903:2-078 2.20$0         438/079082.704990$.080380                  .

0.0.0.1107039#0854380#.90 # !# ! #%  #$! $ %0-099076:.3908 .8 0307.3/ ..8...890 4789                                         .90/ 90 ...0 0770854380 8050.430.890 -08970854380..

  .:89420783945.90/708543809470.03.9.9.4..994050.7.08 %0892....:-041#.08 7057080390/ ..94.3/428.0..:8942078941430243 :/.9..2509..4894.089 70850.-:/099.4 . :8942078$4:/#0.0790892.0.03.48 .9.&83#08940907230.5:8 803/34:9 .

9.024/0  .9089809.08 /70.07 9/408349:80.0890893902.#8.0/43-:/3.908932..570/.43.059 90 .4.99089 8947.%0893094/44 #3.709303.9.709380380 40.0844/2./. $:.7432039 347/079413/90-089.

0 70.:.90-70.03:050770854380 097708543807.3.90 574197  7 57419 7 7 77  2.94.90 ..48950750. 0.250 .03:050770854380 30970.4483!7419.09 70854380390892.3  .03708543807.-0#08 .

943                                         .3/ %080.-008 # !# ! #%  #$! $ 10300/.  708543807.0.03 0 ..4483!7419.08 ..4:39147 4190 545:.9094 -70.4480.

0841 8.4472.78 %030.425.74/8 %0249.30.943.4/0830/94 ..&83#08.84397474:58 '$.:/0/ .3/902.0  .7089.390/949089.147 9003.9.3.98 -.43.94 0...550..389 490781742908.709802039.0.3/4:3.9.89405493489.0841574/:.3/ 8 %090892../:90..20#..

9.8 998920 080094.:3/0789.4/0/ 39880..7090591.:889030:7.3094724/08 .3.-.3/3 4130:7.943 0/8.8..09478147#0854380 4/03 #84:7-.894-0.-.3:2-0741/7.30947 7085438024/0-:9-'$  .8.&830:7.80302094/ #.

703498.-054.-054.0 .09841-.4708 8./  #.!74-028990#5574./708543/0783 44/  #.08  .0.90892.2880825479.08 90702.8845507834.0984144/708543/0783 -.38706:70/ #..440 90.398020398 7892.3.88320398.02-07902..5574. #.9.0-099079.4 90702...0894550/-:31742.

3-0..0:80418420 ...:.5.20.904708 '$348./4083492.-0/.5574.08 9070. '$348070.08 #.07 908.-4:95.703:2-0741#.078  .9:0 #..8 '$348..:8942078.08 9080.:89420784..!74-028990#5574. .3/ 4.98908.:89420788:11071742 411071..47 5:7...89-0.9.7019.94341.:8942078.5574.4393:0/ !74107.0 /0247.808 709:738  .07..205045003/:5390 -089 #.

.087..91:9:70 -0.33/.8838.0 -.90 -0.0480..8 ..809908.4393:0/ 4.9.47410./:..:80.82.7093970..0743041.0/0139438903/94-082589.839490 8.980.9079.:89420794.47820.8:70/ 570/. 98 /4083 9.080380 !70/.70247054071:9.-0894570/.. -0.20.:843.0/0139438.7.:894207  ..70574108 57413.024/08:8035:9.. ..3574108 .20-.70.024/08.80/2.!74-0289#5574.9.

2008803.4:/...438:9../949784209330 '$470/9$$.3//0..36:0 3 9007070.84397008  ..390/990 90.0889#390  8 '$-0.0-09907708:98 %0.94'$ 1907842039.70/# 70708843 30:7. 309478 .8:.0..398948001 39075780307.425.

88:2324/08. '$-:9# 70708843  30:7..3 9024/0809/.30947 .3 .944:/708543/949449072.03:08 .9.9.3/57419 ..9.89 -:9247070.843970024/0894 570/.90050.201742570./-003:80/  .90/70854380 70.083 %024/0809/.:.5072039.843905.94-072.9.3/343 708543/078 &8398/.43708543/078..3//0.4:8.9.38 9.4 2..02-072. .389/.3 4.0399.9070.

935..700.790743-.808174290.9.2039-507843..07....0.:894207 4:804/ :2-07416:...78850395076:.9.70/9.74:8.8947/07 :2-07415:7.83/.0/ ...7/  47-49 :2-0741.90.49508 4.0..43.9.0..485:7.7057080390/  .. ...883.80.78 4909.-0.80/ :2-0741/.9#   .8943047/075..790789.

.9 4/40:0888.8030970850.843970089#-.803/4.9709:73433.090.9.44/08070:/0/ 425.07  .7070708843 30:7.0.08 44.3/ /0.0393.994 507.0883.78. $$.309478 .425.08920393/.3/'$070.70.07.233 5740.78438 54909.

90/:8390# .50398 9908.9:.90 //3 9 .9.5574.98 147.3 899.&838947./4.70 904/08 .9:.0 85039.470  .#0854380.07.3904300307..8:80/948.039..9.4480 9843945 243574850...208.4 .030724/085..08503/34170.24/0.50398 :80../0803907.3/. 4/40:088.984204304:/...4709089./11070392.7885039 147343 70.94425.70..50398 :80.

90/ -'$ %024/0570/..4 %024/00/0/.8990945.3094724/0.%0#08:98 %030:7.58 /0503/34390850.70.087.89024/0800.8038.431/039.9438.309478  .1.089203941   470/09. 09 870.90/.33.0/31472.9..709:73433.33 1742 94  507.4130:7.

 .90.70:8:.09477.07 35:9.4:2381742.9:70 4:95:9 .80  .9.07 //03.07 35:98. /.-.0:7.

0098 90.708:220/.//034/0 4:95:9 35:98. 97.3814720/ 38.3/ %04:95:934/0 1:3..3/ /.943882.7  .7070..9.38/0.880/974:..34/0 090/ 35:98.3/ 5.943       35:98 %04:95:9.381071:3.. 897.:08 -09003 .3:2-078 -09003 .

83035:98.0:7.10/97.33.0947 .3/7.3/42 39.570 850.03.3098 4-807.04305.99073 .33809.%7.

9.943  .42-3. 70./:89903098 89450313.920 907.94385-0900335:98.098 -089 705708039900307.4:95:9.0.9.

47928  .. .9.943 3.9438 %88.0/-.9073..0..42-3.5574.5745.894:800309.4:95:9..

3/570/.95071472.448090-08980941098 1479024/0 425.0/:70147&830:7.943809 .39030:7.7024/08../090/.././..3.7.7.9438098 %7.9035:9.09478 %7.3/4:95:9.0:83  90.!74.9..3/ ..30 &8090908980994.381472.3/ .30947:839097.3/ .-08-09003 .33809 :39900774782320/4790309834 4307.33 90893 .39497.-0894-0-09003  .

-089.7390/0..30.7..894-0.99.3/ 490780  .33809 .70 703907.7.84397001789.84397008..309478.0:7.7079097.8439700390 30:7.7.-09./0..09478!91..084390 .. -:7/038420 43084:943-:/.9047./:890/ 90.550.425:9..30947 .3/:8090 .3/0:3/70/841 35:9.9.-08 0:7.:0190..8 39097.:8942078..33809 /0130.3349 247035:934/08 2470309894-0..943.4/0/ 0.-08 0  70 70390741#/0797:.8.7.

9309479 .70 4:9078 0  3443.8830/ .8830/ /110703.08-09003.80.074300808.3/904200884:/-04-8.381472.:80/-.09478!91.4/0/ 4393:0/ 04390444:9147/.9088.79284786:.9438:834..9.4005741088478  .7/44898 .:70/ 5488-084:94389744:94:9078475071472 97.0:7..894-0.3/0.7074498  .

3..943..8:.3/ 3411 0  .5   .5907 3077..94394..55.9.%4:9843# 7488 8024/08800.400 800.943 #70.

#$4:7.0.0077..02039%74:.9.438:90/..08-04  .33 $$3899:90   .89073.8-003 -47740/ .08 %0.7390808/08   .94385 .36:08 0   -077.33%0.0077.3.479419080#8/08.892.3/47/433411 :894207#0.9.84.3/47/433411 .9.3/3411 35705.. 33 430 $438  ./..590/ 4770470/1742430419094 84:7.

Sign up to vote on this title
UsefulNot useful