Prepared by: Srinivasan 109CE21 Manas 109CE22 Karan 109CE23


Data explosion problem
‡ Automated data collection tools and mature database

technology lead to tremendous amounts of data stored in databases, data warehouses and other information repositories
€ €

We are drowning in data, but starving for knowledge! Solution: Data warehousing and data mining
‡ Data warehousing and on-line analytical processing ‡ Extraction of interesting knowledge (rules, regularities,

patterns, constraints) from data in large databases

€ Lots of data is being collected and warehoused ‡ Web data.g. e-commerce ‡ purchases at department/ grocery stores ‡ Bank/Credit Card transactions € € Computers have become cheaper and more powerful Competitive Pressure is Strong ‡ Provide better. in Customer Relationship Management) . customized services for an edge (e.

€ Data collected and stored at enormous speeds (GB/hour) ‡ remote sensors on a satellite ‡ telescopes scanning the skies ‡ microarrays generating gene expression data ‡ scientific simulations generating terabytes of data € € Traditional techniques infeasible for raw data Data mining may help scientists ‡ in classifying and segmenting data ‡ in Hypothesis Formation .

previously unknown and potentially useful) information or patterns from data in large databases € Alternative names and their ´inside storiesµ: ‡ Data mining: a misnomer? ‡ Knowledge discovery(mining) in databases (KDD). knowledge extraction. data archeology. . implicit. etc.€ Data mining (knowledge discovery in databases): ‡ Extraction of interesting (non-trivial. data/pattern analysis. business intelligence.

€ Decisions in data mining ‡ Kinds of databases to be mined ‡ Kinds of knowledge to be discovered ‡ Kinds of techniques utilized ‡ Kinds of applications adapted € Data mining tasks ‡ Descriptive data mining ‡ Predictive data mining .

clustering. active. association. WWW. stock market analysis. DNA mining. etc. machine learning. heterogeneous. deviation and outlier analysis. Weblog analysis. . trend. telecommunication. Web mining. transactional. time-series. etc. Knowledge to be mined ‡ Characterization. spatial. banking. legacy. object-oriented. fraud analysis. object-relational. discrimination. etc. text. neural network.€ € € € Databases to be mined ‡ Relational. visualization. etc. data warehouse (OLAP). ‡ Multiple/integrated functions and mining at multiple levels Techniques utilized ‡ Database-oriented. classification. statistics. multi-media. Applications adapted ‡ Retail.

Common data mining tasks ‡ ‡ ‡ ‡ ‡ ‡ Classification [Predictive] Clustering [Descriptive] Association Rule Discovery [Descriptive] Sequential Pattern Discovery [Descriptive] Regression [Predictive] Deviation Detection [Predictive] .€ Prediction Tasks ‡ Use some variables to predict unknown or future values of other variables € Description Tasks ‡ Find human-interpretable patterns that describe the data.

‡ Data Analysis is more in line with standard statistical software (ie: web stats). average visit time. etc.‡ In terms of software and the marketing thereof Data Mining != Data Analysis ‡ Data Mining implies software uses some intelligence over simple grouping and partitioning of data to infer new information. These usually present information about subsets and relations within the recorded data set (ie: browser/search engine usage. ) .


and a similarity measure among them. ‡ Other Problem-specific Measures. .€ Given a set of data points. each having a set of attributes. € Similarity Measures: ‡ Euclidean Distance if attributes are continuous. ‡ Data points in separate clusters are less similar to one another. find clusters such that ‡ Data points in one cluster are more similar to one another.

x Find clusters of similar customers. x Measure the clustering quality by observing buying patterns of customers in same cluster vs. . ‡ Approach: x Collect different attributes of customers based on their geographical and lifestyle related information. those from different clusters.€ Market Segmentation: ‡ Goal: subdivide a market into distinct subsets of customers where any subset may conceivably be selected as a market target to be reached with a distinct marketing mix.

. Use it to cluster. ‡ Gain: Information Retrieval can utilize the clusters to relate a new document or search term to clustered documents.€ Document Clustering: ‡ Goal: To find groups of documents that are similar to each other based on the important terms appearing in them. ‡ Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms.

Newspaper. Milk Beer. Bread Beer. Milk} --> {Beer} .€ Given a set of records each of which contain some number of items from a given collection. ‡ Produce dependency rules which will predict occurrence of an item based on occurrences of other items. Newspaper. Newspaper. Milk Beer. TID Items 1 2 3 4 5 Bread. Milk Coke. Coke. Coke. Bread. Milk Rules Discovered: {Milk} --> {Coke} {Newspaper.

« } --> {Coke} ‡ Coke as consequent => Can be used to determine what should be done to boost its sales. ‡ Milk in the antecedent => Can be used to see which products would be affected if the store discontinues selling milk. ‡ Milk in antecedent and Coke in consequent => Can be used to see what products should be sold with milk to promote sale of coke! .€ Marketing and Sales Promotion: ‡ Let the rule discovered be {Milk.

‡ A classic rule -x If a customer buys Newspaper and milk.€ Supermarket shelf management. ‡ Goal: To identify items that are bought together by sufficiently many customers. then he is very likely to buy beer: . ‡ Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items.


significant deviations from normal behavior € Applications: € Detect ‡ Credit Card Fraud Detection ‡ Network Intrusion Detection .

. All horses are mammals 2. Therefore. All mammals have lungs 3. all horses have lungs. Therefore. all horses have lungs € Induction reasoning adds information: 1. 2. All horses observed so far have lungs.Induction vs Deduction € Deductive reasoning is truth-preserving: 1.

Pattern Evaluation ‡ Data mining: the core of knowledge discovery process. Data Selection Data Preprocessing Data Mining Task-relevant Data Data Warehouse Data Cleaning Data Integration Databases .

dimensionality/variable reduction. invariant representation. € € € Choosing the mining algorithm(s) Data mining: search for patterns of interest Pattern evaluation and knowledge presentation ‡ visualization. regression. clustering. € Use of discovered knowledge . € Choosing functions of data mining ‡ summarization. classification.€ € € € Learning the application domain: ‡ relevant prior knowledge and goals of application Creating a target data set: data selection Data cleaning and preprocessing: (may take 60% of effort!) Data reduction and transformation: ‡ Find useful features. transformation. etc. removing redundant patterns. association.

incorporation of domain knowledge not possible ‡ Interpretation of results is difficult and daunting ‡ Requires expert user guidance Data Mining: ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Large Data sets Efficiency of Algorithms is important Scalability of Algorithms is important Real World Data Lots of Missing Values Pre-existing data .not user generated Data not static .prone to updates Efficient methods for data retrieval available for use .Statistical Analysis: ‡ Ill-suited for Nominal and Structured Data Types ‡ Completely data driven .

preferences. and thus developing intelligent AI opponents. ‡ Risk Analysis Product Defect Analysis Analyze product defect rates for given plants and predict possible complications (read: lawsuits) down the line. (ie: Chess) ‡ Business Strategies Market Basket Analysis Identify customer demographics. .‡ AI/Machine Learning Combinatorial/Game Data Mining Good for analyzing winning strategies to games. and purchasing patterns.

. Similarly.‡ User Behavior Validation Fraud Detection In the realm of cell phones Comparing phone activity to calling records. comparing purchases with historical purchases. Can detect activity with stolen cards. with credit cards. Can help detect calls made on cloned phones.