DATA MINING AND WAREHOUSING

Presented by : P.Satya vathi M.Divya SRI SIVANI COLLEGE OF ENGINEERING SRIKAKULAM EMAIL ID : vathi.satya2@gmail.com divya.moyya@gmail.com usually contains historical data derived from transaction data, but

ABSTRACT..
Data mining is a combination of database and artificial intelligence technologies. Although the AI field has taken a major dive in the last decade; this new emerging field has shown that AI can add major contributions to existing fields in computer science. In fact, many experts believe that data mining is the third hottest field in the industry behi-nd the Internet, and data warehousing. Data mining is really just the next step in the process of analyzing data. Instead of getting queries on standard or user-specified relationships, data mining goes a step farther by finding meaningful relationships in data. Relationships that were thought to have not existed or ones that give a more insightful view of the data. For example, a computer-generated graph may not give the user any insight; however data mining can find trends in the same data that shows the user more precisely what is going on. Using trends that the end-user would have never thought to query the computer about. A data warehouse is a repository of an organization's electronically stored data. Data warehouses are designed to facilitate reporting and analysis.This classic definition of the data warehouse focuses on data storage. However, the means to retrieve and analyze data, to extract, transform and load data, and to manage the dictionary data are also considered essential components of a data warehousing system. Many references to data warehousing use this broader context. Thus, an expanded definition for data warehousing includes business intelligent tools to extract, transform, and load data into the repository, and tools to manage and retrieve metadata A data warehouse is a relational database that is designed for query and analysis rather than for transaction processing. It can include data from other sources. Data warehouses separate analysis workload from transaction workload and enable an organization to consolidate data from several sources. This helps in:

Maintaining historical records Analyzing the data to gain a better understanding of the business and to improve the business.

Introduction :Data mining, the extraction of hidden predictive information from large databases, is a powerful new technology with great potential to help companies focus on the most important information in their data warehouses. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions. Data mining tools can answer business questions that traditionally were too time consuming to resolve. This evolution began when business data was first stored on computers, continued with improvements in data access, and more recently, generated technologies that allow users to navigate through their data in real time. Massive data collection

Powerful multiprocessor computers Data mining algorithms

The Scope of Data Mining Data mining derives its name from the similarities between searching for valuable business information in a large database for example,

finding linked products in gigabytes of store scanner data and mining a mountain for a vein of valuable ore. Both processes require either sifting through an immense amount of material, or intelligently probing it to find exactly where the value resides.

layer in this case has two nodes, Z1 and Z2 representing output values we are trying to determine from the inputs. 3.2.1 Decision trees Decision trees are simple knowledge representation and they classify examples to a finite number of classes, the nodes are labeled with attribute names, the edges are labeled with possible values for this attribute and the leaves labeled with different classes. The following is an example of objects that describe the weather at a given time. The objects contain information on the outlook, humidity etc. Some objects are positive examples denote by P and others are negative i.e. N.

Automated prediction of trends and behaviors. Data mining automates the process of finding predictive information in large databases. Questions that traditionally required extensive hands-on analysis can now be answered directly from the data. Automated discovery of previously unknown patterns. Data mining tools sweep through databases and identify previously hidden patterns in one step. An example of pattern discovery is the analysis of retail sales data to identify seemingly unrelated products that are often purchased together.

Techniques: Decision tree structure 3.3 Neural networks Neural networks have broad applicability to real world business problems and have already been successfully applied in many industries. Since neural networks are best at identifying patterns or trends in data, they are well suited for prediction or forecasting needs including:

Genetic algorithms: Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of evolution. Nearest neighbor method: A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called the knearest neighbor technique. Rule induction: The extraction of useful ifthen rules from data based on statistical significance.

sales forecasting industrial process control customer research data validation risk management target marketing etc.

The bottom layer represents the input layer, in this case with 5 inputs labels X1 through X5. In the middle is something called the hidden layer, with a variable number of nodes. It is the hidden layer that performs much of the work of the network. The output

Customers General information (e.g. demographic data) Proprietary information (e.g. customer transactions) Known

Prospects Known

Known

Target

:-How Data Mining Works How exactly is data mining able to tell you important things that you didn't know or what is going to happen next? The technique that is used to perform these feats in data mining is called modeling. For instance, if you were looking for a sunken Spanish galleon on the high seas the first thing you might do is to research the times when Spanish treasure had been found by others in the past. You might note that these ships often tend to be found off the coast of Bermuda and that there are certain characteristics to the ocean currents, and certain routes that have likely been taken by the ships captains in that era. You note these similarities and build a model that includes the characteristics that are common to the locations of these sunken treasures. With these models in hand you sail off looking for treasure where your model indicates it most likely might be given a similar situation in the past. This act of model building is thus something that people have been doing for a long time, certainly before the advent of computers or data mining technology. What happens on computers, however, is not much different than the way people build models. Computers are loaded up with lots of information about a variety of situations where an answer is known and then the data mining software on the computer must run through that data and distill the characteristics of the data that should go into the model. For example, say that you are the director of marketing for a telecommunications company and you'd like to acquire some new long distance phone customers. Table 2 - Data Mining for Prospecting

Table 3 shows another common scenario for building models: predict what is going to happen in the future.

Yesterday Static information and current Known

Today Known

Tomorrow Known

plans (e.g. demographic data, marketing plans)

Dynamic information (e.g. customer transactions)

Known

Known

Target The ideal starting point is a data warehouse containing a combination of internal data tracking all customer contact coupled with external market data about competitor activity. Background information on potential customers also provides an excellent basis for prospecting. :-Applications A wide range of companies have deployed successful applications of data mining. Combating Terrorism Data mining has been cited as the method by which the U.S. Army unit Able Danger had identified the September 11, 2001 attacks leader, Mohamed Attar, and three other 9/11 hijackers as possible members of an Al Qaeda cell operating in the U.S. more than a year before the attack.

Table 3 - Data Mining for Predictions

Architecture for Data Mining : To best apply these advanced techniques, they must be fully integrated with a data warehouse as well as flexible interactive business analysis tools. The resulting analytic data warehouse can be applied to improve business processes throughout the organization, in areas such as promotional campaign management, fraud detection, new product rollout, and so on. Figure 1 illustrates architecture for advanced analysis in a large data warehouse.

A pharmaceutical company can analyze its recent sales force activity and their results to improve targeting of high-value physicians and determine which marketing activities will have the greatest impact in the next few months. A credit card company can leverage its vast warehouse of customer transaction data to identify customers most likely to be interested in a new credit product. Using a small test mailing, the attributes of customers with an affinity for the product can be identified. A diversified transportation company with a large direct sales force can apply data mining to identify the best prospects for its services. A large consumer package goods company can apply data mining to improve its sales process to retailers.

Introduction: Most firms want to set up transaction processing systems so there is a high probability that transactions will be completed in what is judged to be an acceptable amount of time. Reports and queries, which can require a much greater range of limited server/disk resources than

Figure 1 - Integrated Data Mining Architecture

transaction processing, run on the servers/disks used by transaction processing systems can lower the probability that transactions complete in an acceptable amount of time. Or, running queries and reports, with their variable resource requirements, on the servers/disks used by transaction processing systems can make it quite complex to manage servers/disks so there is a high enough probability that acceptable response time can be achieved. Definition: Data Warehouse: The term Data Warehouse was coined by Bill Inman in 1990, which he defined in the following way: "A warehouse is a subject-oriented, integrated, timevariant and non-volatile collection of data in support of management's decision making process". He defined the terms in the sentence as follows: Subject Oriented: Data that gives information about a particular subject instead of about a company's ongoing operations. Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole. Time-variant: All data in the data warehouse is identified with a particular time period. Non-volatile Data is stable in a data warehouse. More data is added but data is never removed. This enables management to gain a consistent picture of the business.

Data Warehouse Architecture: with a Staging Area and Data Marts

Data Warehouse Architecture: Basic Figure 1-2 shows a simple architecture for a data warehouse. End users directly access data derived from several source systems through the data warehouse. Figure 1-2 Architecture of a Data Warehouse

Description of "Figure 1-2 Architecture of a Data Warehouse" In Figure 1-2, the metadata and raw data of a traditional OLTP system is present, as is an additional type of data, summary data. Data

Warehouse Architecture: with a Staging Area You need to clean and process your operational data before putting it into the warehouse, as shown in Figure 1-2.

Data Warehouse Architectures Data warehouses and their architectures vary depending upon the specifics of an organization's situation. Three common architectures are:

Data Warehouse Architecture: Basic Data Warehouse Architecture: with a Staging Area

Figure 1-3 Architecture of a Data Warehouse with a

St aging Area

Description of "Figure 1-3 Architecture of a Data Warehouse with a Staging Area" Data Warehouse Architecture: with a Staging Area and Data Marts Although the architecture in Figure 1-3 is quite common, you may want to customize your warehouse's architecture for different groups within your organization. Figure 1-4 Architecture of a Data Warehouse with a Staging Area and Data Marts

Pilot projects or projects requiring rapid prototyping and rapid time-to-value Off-loading projects from the enterprise data warehouse; ie large analytical query projects that affect the overall workload of the enterprise data warehouse Applications with specific performance or loading requirements Data marts that have outgrown their present environment Turnkey data warehouses or data marts Solutions for applications with high data growth and high performance requirements Applications requiring data warehouse encryption

Disadvantages of data warehouses There are also disadvantages to using a data warehouse. Some of them are:

Description of "Figure 1-4 Architecture of a Data Warehouse with a Staging Area and Data Marts" Note:

Over their life, data warehouses can have high costs. The data warehouse is usually not static. Maintenance costs are high. Data warehouses can get outdated relatively quickly. There is a cost of delivering suboptimal information to the organization. There is often a fine line between data warehouses and operational systems. Duplicate, expensive functionality may be developed. Or, functionality may be developed in the data warehouse that, in retrospect, should have been developed in the operational systems and vice versa..

The future of data warehousing Data marts are an important part of many data warehouses, but they are not the focus of this book. Data warehouse Components Data warehousing is essentially what you need to do in order to create a data warehouse, and what you do with it. It is the process of creating, populating, and then querying a data warehouse and can involve a number of discrete technologies Application Uses DW appliances provide solutions for many analytic application uses, including: Data warehousing, like any technology niche, has a history of innovations that did not receive market acceptance.

Service Oriented Architecture Search capabilities integrated into reporting and analysis technology Software as a Service Analytic tools that work in memory Visualization

Another prediction is that data warehouse performance will continue to be improved by use of data warehouse appliances,

Enterprise data warehousing Super-sized sandboxes isolate power users with resource intensive queries

Difference between data mining and data ware housing :

6. DATA WAREHOUSING & FUNDAMENTALS

Data mining: A method of comparing large amounts of data to find patters. Normally this is used for models and forecasting. Or The process of discovering meaningful correlations, patterns, and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as statistical and mathematical techniques. Data warehousing: The ability of a system to store data resulting from Data Mining to be used in future inquiries of that database. Or A data warehouse is a central repository (or storehouse) for data that an enterprise's various business systems collect. Data from various online applications and other sources is selectively extracted and organized in the data warehouse for useful analysis

Conclusion : Comprehensive data warehouses that integrate operational data with customer, supplier, and market information have resulted in an explosion of information. Competition requires timely and sophisticated analysis on an integrated view of the data. However, there is a growing gap between more powerful storage and retrieval systems and the users ability to effectively analyze and act on the information they contain. A new technological leap is needed to structure and prioritize information for specific end-user problems. The data mining tools can make this leap. Quantifiable business benefits have been proven through the integration of data mining with current information systems, and new products are on the horizon that will bring this integration to an even wider audience of users. REFERENCES : 1.DATA MINING TECHNIQUES BY ARUN K
PUJARI

2. DATA WAREHOUSING IN THE REAL WORLD BY SAM ANAHORY & DENNIS MURRY 3. GOOGLE SEARCH & MSN SEARCH 4. ENCYLOPEDIA 5. IEEE magazines