Access XLMiner

Access XLMiner through Citrix

Install the Citrix client on your laptop: Go to http://www.mcmaster.ca/uts/vip/citrix/ins and follow the instructions

Access XLMiner

Access the Citrix log-in page: https://macapps.mcmaster.ca and use your mac email and password to login Click on the RJC > Microsoft Office folder and then click on Excel Miner 3_3. xLMiner menu appears in the Addins tab on Excel.

Access XLMiner

The folder that appears by default is My Documents. This folder is not the one in your local computer, so you have to browse in the Look in option in the top of the dialog box in order to find your files. Your local drives will appear with a format similar to C$ on Client (V:). When you click in your drive, a new dialog box will show up. In this dialog box, you will be asked What access do you want to grant?. Please select the option Full access. It will give you the possibility to modify and save the files you have in your computer.

Data Preparation and Exploration

Sampling

Obtaining Data: Sampling

Data mining typically deals with huge databases Algorithms and models are typically applied to a sample from a database, to produce statistically-valid results XLMiner, e.g., limits the training partition to 10,000 records Once you develop and select a final model, you use it to score the observations in the larger database

Rare event oversampling

Often the event of interest is rare Examples: response to mailing, fraud in taxes, Sampling may yield too few interesting cases to effectively train a model A popular solution: oversample the rare cases to obtain a more balanced training set Later, need to adjust results for the oversampling

Pre-processing Data

Types of Variables

Determine the types of pre-processing needed, and algorithms used Main distinction: Categorical vs. numeric

Numeric
Continuous Integer Categorical Ordered (low, medium, high) Unordered (male, female)

Variable handling

Numeric

Most algorithms in XLMiner can handle numeric data May occasionally need to bin into categories Nave Bayes can use as-is In most other algorithms, must create binary dummies (number of dummies = number of categories 1)

Categorical

Detecting Outliers

An outlier is an observation that is extreme, being distant from the rest of the data (definition of distant is deliberately vague) Outliers can have disproportionate influence on models (a problem if it is spurious) An important step in data preprocessing is detecting outliers Once detected, domain knowledge is

Examples of Outliers: Guinness World Records

Svetlana Pankratova, world's leggiest woman (at 52"), and He Pingping, the world's shortest man (at 2' 5.37").

Sultan Kosen from Turkey, is 2m 46.5cm (8ft 1in) tall

Detecting Outliers

In some contexts, finding outliers is the purpose of the DM exercise (airport security screening). This is called anomaly detection.

Handling Missing Data

Most algorithms will not process records with missing values. Default is to drop those records. Solution 1: Omission

If a small number of records have missing values, can omit them If many records are missing values on a small set of variables, can drop those variables (or use proxies) If many records have missing values, omission is not practical Replace missing values with reasonable substitutes

Solution 2: Imputation

Normalizing (Standardizing) Data

Used in some techniques when variables with the largest scales would dominate and skew results Puts all variables on same scale Normalizing function: Subtract mean and divide by standard deviation (used in XLMiner) Alternative function: scale to 0-1 by subtracting minimum and dividing by the range

Partitioning the Data

Partitioning the Data

Problem: How well will our model perform with new data? Solution: Separate data into two parts

Training partition to develop the model Validation partition to implement the model and evaluate its performance on new data

Test Partition

When a model is developed on training data, it can overfit the training data (hence need to assess on validation) Assessing multiple models on same validation data can overfit validation data Some methods use the validation data to choose a parameter. This too can lead to overfitting the validation data Solution: final selected model is applied to a test partition to give

Using Excel and XLMiner for Data Mining

Excel is limited in data capacity However, the training and validation of DM models can be handled within the modest limits of Excel and XLMiner Models can then be used to score larger databases XLMiner has functions for interacting with various databases (taking samples from a database, and scoring a database from a developed model)

http://www.resample.com/xlminer/help/Index.htm

Boston House Median Value Prediction

C R I M p e r c a p ita c rim e r a te b y to w n Z N p r o p o r tio n o fr e s id e n tia lla n d z o n e d fo rlo ts o v e r2 5 ,0 0 0 s q .ft. I N D U S p r o p o r tio n o fn o n -r e ta ilb u s in e s s a c r e s p e rto w n . C H A S C h a r le s R iv e rd u m m y v a r ia b le (1 iftr a c tb o u n d s r iv e r;0 o th e rw is e ) N O X n itric o x id e s c o n c e n tra tio n (p a r ts p e r1 0 m illio n ) R M a v e r a g e n u m b e r o fro o m s p e rd w e llin g A G E p r o p o rtio n o fo w n e r-o c c u p ie d u n its b u iltp r io r to 1 9 4 0 D I S w e ig h te d d is ta n c e s to fiv e B o s to n e m p lo y m e n tc e n tr e s R A D in d e x o fa c c e s s ib ility to ra d ia lh ig h w a y s T A X fu ll-v a lu e p r o p e rty -ta x ra te p e r $ 1 0 ,0 0 0 P T R A T I O p u p il-te a c h e rra tio b y to w n B 1 0 0 0 (B k -0 .6 3 )^ 2 w h e re B k is th e p r o p o rtio n o fb la c k s b y to w n L S T A T % lo w e rs ta tu s o fth e p o p u la tio n M E D V M e d ia n v a lu e o fo w n e r -o c c u p ie d h o m e s in $ 1 0 0 0

Example Linear Regression Boston Housing Data

A CRIM 0.006 0.027 0.027 0.032 0.069 0.030 0.088 0.145 0.211 0.170 B C D E NOX 0.54 0.47 0.47 0.46 0.46 0.46 0.52 0.52 0.52 0.52 F RM 6.58 6.42 7.19 7.00 7.15 6.43 6.01 6.17 5.63 6.00 G AGE 65.2 78.9 61.1 45.8 54.2 58.7 66.6 96.1 100 85.9 H I J K L M B LSTAT 397 397 393 395 397 394 396 397 387 387 5 9 4 3 5 5 12 19 30 17 N O ZN INDUS CHAS 18 0 0 0 0 0 12.5 12.5 12.5 12.5 2.31 7.07 7.07 2.18 2.18 2.18 7.87 7.87 7.87 7.87 0 0 0 0 0 0 0 0 0 0 DIS RAD 4.09 4.97 4.97 6.06 6.06 6.06 5.56 5.95 6.08 6.59 1 2 2 3 3 3 5 5 5 5 TAX PTRATIO 296 242 242 222 222 222 311 311 311 311 15.3 17.8 17.8 18.7 18.7 18.7 15.2 15.2 15.2 15.2 CAT. MEDV MEDV 24 21.6 34.7 33.4 36.2 28.7 22.9 27.1 16.5 18.9 0 0 1 1 1 0 0 0 0 0

Partitioning the data

Using XLMiner for Multiple Linear Regression: Specify input and output var.

Specifying Output

RMS error

Error = actual - predicted RMS = Root-mean-squared error = Square root of average squared error In previous example, sizes of training and validation sets differ, so only RMS Error and Average Error are comparable

Summary

Data Mining consists of supervised methods (Classification & Prediction) and unsupervised methods (Association Rules, Data Reduction, Data Exploration & Visualization) Before algorithms can be applied, data need to be characterized and preprocessed To evaluate performance and to avoid overfitting, data partitioning is used Data mining methods are usually applied to

Data Exploration and Dimension Reduction

Exploring the data

Statistical summary of data: common metrics

Average (sample mean) Median (the middle point) Minimum Maximum Standard deviation Counts & percentages

Boston House Data

C R I M p e r c a p ita c rim e r a te b y to w n Z N p r o p o r tio n o fr e s id e n tia lla n d z o n e d fo rlo ts o v e r2 5 ,0 0 0 s q .ft. I N D U S p r o p o r tio n o fn o n -r e ta ilb u s in e s s a c r e s p e rto w n . C H A S C h a r le s R iv e rd u m m y v a r ia b le (1 iftr a c tb o u n d s r iv e r;0 o th e rw is e ) N O X n itric o x id e s c o n c e n tra tio n (p a r ts p e r1 0 m illio n ) R M a v e r a g e n u m b e r o fro o m s p e rd w e llin g A G E p r o p o rtio n o fo w n e r-o c c u p ie d u n its b u iltp r io r to 1 9 4 0 D I S w e ig h te d d is ta n c e s to fiv e B o s to n e m p lo y m e n tc e n tr e s R A D in d e x o fa c c e s s ib ility to ra d ia lh ig h w a y s T A X fu ll-v a lu e p r o p e rty -ta x ra te p e r $ 1 0 ,0 0 0 P T R A T I O p u p il-te a c h e rra tio b y to w n B 1 0 0 0 (B k -0 .6 3 )^ 2 w h e re B k is th e p r o p o rtio n o fb la c k s b y to w n L S T A T % lo w e rs ta tu s o fth e p o p u la tio n M E D V M e d ia n v a lu e o fo w n e r -o c c u p ie d h o m e s in $ 1 0 0 0

Summary Statistics Boston Housing

Correlations Between Pairs of Variables:

Correlation Matrix from Excel
PTRATIO B PTRATIO 1 B -0.17738 1 LSTAT 0.374044 -0.36609 MEDV -0.50779 0.333461 LSTAT MEDV

1 -0.73766

PTRATIO pupil-teacher ratio by town B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town LSTAT % lower status of the population MEDV Median value of owner-occupied homes in $1000's

Summarize Using Pivot Tables

Counts & percentages are useful for summarizing categorical data Boston Housing example: 471 neighborhoods border the Charles River (1) 35 neighborhoods do not (0)
Count of MEDV CHAS Total 0 471 1 35 Grand Total 506

Pivot Tables - cont.

Averages are useful for summarizing grouped numerical data

In Boston Housing example: Compare average home values in neighborhoods that border Charles River (1) and those that do not (0)

Average of MEDV CHAS Total 0 1 Grand Total

22.09 28.44 22.53

Pivot Tables, cont.

Group by multiple criteria: By # rooms and location E.g., neighborhoods on the Charles with 6-7 rooms have average house value of 25.92 ($000)
Average of MEDV CHAS RM 0 3-4 25.30 4-5 16.02 5-6 17.13 6-7 21.77 7-8 35.96 8-9 45.70 Grand Total 22.09 1 Grand Total 25.30 16.02 22.22 17.49 25.92 22.02 44.07 36.92 35.95 44.20 28.44 22.53

Graphs

Histograms
Boston Housing example:
180

H istog ram
160 140 120 100 80 60 40 20 0 5 10 15 20 25 30 35 40 45 50 ME D V

Histogram shows the distribution of the outcome variable (median house value)
Mode: the value with highest frequency

y c n u q e r F

Box Plot
outliers max

Quartile 3 mean Median Quartile 1 min outlier

Top outliers defined as those above Q3+1.5(Q3-Q1). max is the maximum of nonoutliers Analogous definitions for bottom outliers and for min Details may differ

Boxplots
B oxPlot
60 50 40 30

Side-by-side boxplots are useful for comparing subgroup

s e u l a V Y

MEDV

20 10 0
0

CH AS

Boston Housing Example: Display distribution of outcome variable (MEDV) for neighborhoods on Charles (1) and not on Charles (0)

Correlation Analysis

Below: Correlation matrix for portion of Boston Housing data Shows correlation between variable pairs
CRIM ZN INDUS CHAS NOX RM 1 -0.20047 1 0.406583 -0.53383 1 -0.05589 -0.0427 0.062938 1 0.420972 -0.5166 0.763651 0.091203 -0.21925 0.311991 -0.39168 0.091251

CRIM ZN INDUS CHAS NOX RM

1 -0.30219

Matrix Plot
Shows scatterplots for variable pairs Example: scatterplots for 3 Boston Housing variables

Reducing Categories

A single categorical variable with m categories is typically transformed into m-1 dummy variables Each dummy variable takes the values 0 or 1

0 = no for the category 1 = yes

Problem: Can end up with too many variables Solution: Reduce by combining categories that are close to each other Use pivot tables to assess outcome variable sensitivity to the dummies Exception: Nave Bayes can handle categorical variables without transforming them into

Principal Components Analysis

Principal Components Analysis

Goal: Reduce a set of numerical variables. The idea: Remove the overlap of information between these variable. [Information is measured by the sum of the variances of the variables.] Final product: a smaller number of numerical variables that contain

Principal Components Analysis

How does PCA do this? Create new variables that are linear combinations of the original variables (i.e., they are weighted averages of the original variables). These linear combinations are uncorrelated (no information overlap), and only a few of them contain most of the original information. The new variables are called principal

Example Breakfast Cereals

nam e m fr 100% _Bran N 100% _Natural_Bran Q All-Bran K All-Bran_with_Extra_FiberK Alm ond_Delight R Apple_Cinnam on_Cheerios G Apple_Jacks K Basic_4 G Bran_Chex R Bran_Flakes P Cap'n'Crunch Q Cheerios G Cinnam on_Toast_Crunch G type calories protein C 70 4 C 120 3 C 70 4 C 50 4 C 110 2 C 110 2 C 110 2 C 130 3 C 90 2 C 90 3 C 120 1 C 110 6 C 120 1 rating 68 34 59 94 34 30 33 37 49 53 18 51 20

Description of Variables

Name: name of cereal mfr: manufacturer type: cold or hot calories: calories per serving protein: grams fat: g. sodium: mg. fiber: g.

carbo: g. complex carbohydrates sugars: g. potass: mg. vitamins: % FDA rec shelf: display shelf weight: oz. 1 serving cups: in one

Consider calories & ratings

c a lo rie s ra tin g s c a lo rie s 3 7 9 .6 3 -1 8 9 .6 8 ra tin g s -1 8 9 .6 8 1 9 7 .3 2

Total variance (=information) is sum of individual variances: 379.63 + 197.32=576.95 Calories accounts for 379.63/576.95 = 66% of total variances

First & Second Principal Components

Z1 and Z2 are two linear combinations Z1 has the highest variation (spread of values) Z2 has the lowest variation

PCA output for these 2 variables

Top: weights to project original data onto z1 & z2 e.g., col. 1 scores are computed z1 scores using weights (0.847, 0.532) Bottom: reallocated variance for new variables

Components Variable calories rating Variance Variance% Cum% P-value 1 -0.84705347 0.53150767 498.0244751 86.31913757 86.31913757 0 2 0.53150767 0.84705347 78.932724 13.68086338 100 1

Properties of the resulting variables

New distribution of information:

New variances = 498 (for z1) and 79 (for z2) Sum of variances = sum of variances for original variables calories and ratings New variable z1 has most of the total variance, might be used as proxy for both calories and ratings z1 and z2 have correlation of zero (no information overlap)

Generalization
X1, X2, X3, Xp, original p variables Z1, Z2, Z3, Zp, weighted averages of original variables All pairs of Z variables have 0 correlation Order Zs by variance (z1 largest, Zp smallest) Usually the first few Z variables contain most of the information, and so the rest can be dropped.

PCA on full data set

Variable calories protein fat sodium fiber carbo sugars potass vitamins shelf weight cups rating Variance Variance% Cum% 1 0.07624155 -0.00146212 -0.00013779 0.98165619 -0.00479783 0.01486445 0.00398314 -0.119053 0.10149482 -0.00093911 0.0005016 0.00047302 -0.07615706 7204.161133 55.52834702 55.52834702 2 -0.01066097 0.00873588 0.00271266 0.12513085 0.03077993 -0.01731863 -0.00013545 0.98861349 0.01598651 0.00443601 0.00098829 -0.00160279 0.07254035 4833.050293 37.25226212 92.78060913 3 0.61074823 0.00050506 0.01596125 -0.14073193 -0.01684542 0.01272501 0.09870714 0.03619435 0.7074821 0.01267395 0.00369807 0.00060208 -0.30776858 498.4260864 3.84177661 96.62238312 4 -0.61706442 0.0019389 -0.02595884 -0.00293341 0.02145976 0.02175146 -0.11555841 -0.042696 0.69835609 0.00574066 -0.0026621 0.00095916 0.33866307 357.2174377 2.75336623 99.37575531 5 0.45754826 0.05533375 -0.01839438 0.01588042 0.00872434 0.35580006 -0.29906386 -0.04644227 -0.02556211 -0.00823057 0.00318591 0.00280366 0.75365263 72.47863007 0.55865192 99.93440247 6 0.12601775 0.10379469 -0.12500292 0.02245871 0.271184 -0.56089228 0.62323487 -0.05091622 0.01341988 -0.05412053 0.00817035 -0.01087413 0.41805118 4.33980322 0.0334504 99.96785736

First 6 components shown First 2 capture 93% of the total variation

Note: data differ slightly from text

Normalizing data

In these results, sodium dominates first PC Just because of the way it is measured (mg), its scale is greater than almost all other variables Hence its variance will be a dominant component of the total variance Normalize each variable to remove scale effect

Divide by std. deviation (may subtract mean first)

PCA using standardized variables

Variable calories protein fat sodium fiber carbo sugars potass vitamins shelf weight cups rating Variance Variance% Cum% 1 0.32422706 -0.30220962 0.05846959 0.20198308 -0.43971062 0.17192839 0.25019819 -0.3834067 0.13955688 -0.13469705 0.07780685 0.27874646 -0.45326898 3.59530377 27.65618324 27.65618324 2 0.36006299 0.16462311 0.34051308 0.12548573 0.21760374 -0.18648526 0.3434512 0.32790738 0.16689315 0.27544045 0.43545634 -0.24295618 -0.22710647 3.16411042 24.3393116 51.99549484 3 0.13210163 0.2609871 -0.21144024 0.37701431 0.07857864 0.56368077 -0.34577203 0.08459517 0.38407779 0.01791886 0.27536476 0.14065795 0.18307236 1.86585701 14.35274601 66.34824371 4 0.30780381 0.43252215 0.37964511 -0.16090299 -0.10126047 0.20293142 -0.10401795 0.00463834 -0.52358848 -0.4340663 0.10600897 0.08945525 0.06392702 1.09171081 8.39777565 74.74601746 5 0.08924425 0.14542894 0.44644874 -0.33231756 -0.24595702 0.12910619 -0.27725372 -0.16622125 0.21541923 0.59693497 -0.26767638 0.06306333 0.03328028 0.96962351 7.45864248 82.20465851 6 -0.20683768 0.15786675 0.40349057 0.6789462 0.06016004 -0.25979191 -0.20437138 0.022951 0.03514972 -0.12134896 -0.38367996 0.06609894 -0.16606605 0.72342771 5.5648284 87.76948547

First component accounts for smaller part of variance Need to use more components to capture same

PCA in Classification/Prediction

Apply PCA to training data Decide how many PCs to use Use variable weights in those PCs with validation/new data This creates a new reduced set of predictors in validation/new data

Summary

Data summarization is important for data exploration Data summaries include numerical metrics (average, median, etc.) and graphical summaries Data reduction is useful for compressing the information in the data into a smaller subset Categorical variables can be reduced by combining similar categories Principal components analysis transforms an original set of numerical data into a smaller set of weighted averages of the original data that contain most of the original information in less variables.