You are on page 1of 8

An IT Mini project

ON Data warehouse
The concept & its emerging trend

Submitted By
Abhishek Kumar Abhishek Kumar Singh Ahmad nawaz Akhouri vivek sinha

Submitted To Mr. Soumen saha Master School Of Management

Meerut .

cleaning and integrating new data from "data marts" that were tailored for ready access by users. usually from long-term existing operational systems (usually referred to as legacy systems). cleaning and integrating data from various sources.Data warehouse A data warehouse (DW) is a database used for reporting. market research and decision support (Marakas & OBrien 2009). BACKGROUND The concept of data warehousing dates back to the late 1980s when IBM researchers Barry Devlin and Paul Murphy developed the "business data warehouse". to extract. Moreover. The access layer is for getting data out for users. Though each environment served different users. Many references to data warehousing use this broader context. and access. A data warehouse maintains its functions in three layers: staging. transform and load data. transform and load data into the repository. Often new requirements necessitated gathering. The integration layer is used to integrate data and to have a level of abstraction from users. In essence. the operational systems were frequently reexamined as new decision support requirements emerged. In larger corporations it was typical for multiple decision support environments to operate independently. The data may pass through an operational data store for additional operations before it is used in the DW for reporting. . This definition of the data warehouse focuses on data storage. an enormous amount of redundancy was required to support multiple decision support environments. an expanded definition for data warehousing includes business intelligence tools. and tools to manage and retrieve metadata.[2] 1970s — ACNielsen and IRI provide dimensional data marts for retail sales. Thus. and to manage the data dictionary are also considered essential components of a data warehousing system. online analytical processing. catalogued and made available for use by managers and other business professionals for data mining. integration. they often required much of the same stored data. In the absence of a data warehousing architecture. mainly the high costs associated with it. the means to retrieve and analyze data. the data warehousing concept was intended to provide an architectural model for the flow of data from operational systems to decision support environments. Key developments in early years of data warehousing were: 1960s — General Mills and Dartmouth College. develop the terms dimensions and facts. Staging is used to store raw data for use by developers (analysis and support). The process of gathering. transformed.[2] 1970s — Bill Inmon begins to define and discuss the term: Data Warehouse 1983 — Teradata introduces a database management system specifically designed for decision support. The main source of the data is cleaned. was typically in part replicated for each environment. The concept attempted to address the various problems associated with this flow. However. tools to extract. The data is offloaded from the operational systems for reporting. in a joint research project.

The InmonKimball differences about design methodology. 1991 — Prism Solutions introduces Prism Warehouse Manager. is founded. Metadata layer The data dictionary — This is usually more detailed than an operational system data dictionary.1988 — Barry Devlin and Paul Murphy publish the article An architecture for a business and information systems in IBM Systems Journal where they introduce the term "business data warehouse". Architecture Operational database layer The source data for the data warehouse — An organization's Enterprise Resource Planning systems fall into this layer. 1996 — Ralph Kimball publishes the book The Data Warehouse Toolkit. 1991 — Bill Inmon publishes the book Building the Data Warehouse. have to do with this layer . Business intelligence tools fall into this layer. a for-profit organization that promotes data warehousing. transform. software for developing a data warehouse. load data into the warehouse fall into this layer. Data access layer The interface between the operational and informational access layer — Tools to extract. 2000 — Daniel Linstedt releases the Data Vault. 1990 — Daniel Linstedt begins work on Developing the Data Vault model and methodology for data warehouses 1990 — Red Brick Systems introduces Red Brick Warehouse. There are dictionaries for the entire warehouse and sometimes dictionaries for the data that can be accessed by a particular reporting and analysis tool. Informational access layer The data accessed for reporting and analyzing and the tools for reporting and analyzing data — This is also called the data mart. discussed later in this article. 1995 — Daniel Linstedt adds SEI/CMMI and Six Sigma to the Data Vault Methodology to manage projects in data warehousing. enabling real time auditable Data Warehouses. 1995 — The Data Warehousing Institute. a database management system specifically for data warehousing.

Ralph 2008). Tables are grouped together by subject areas that reflect general data categories (e. Furthermore. The normalized structure divides data into entities. which are generally numeric transaction data. finance. and It is difficult to modify the data warehouse structure if the organization adopting the dimensional approach changes the way in which it does business.). whose supporters are referred to as ―Kimballites‖. Ralph 2008). also called the 3NF model. . Facts are related to the organization’s business processes and operational system whereas the dimensions surrounding them contain context about the measurement (Kimball. Also. believe in Bill Inmon's approach in which it is stated that the data warehouse should be modeled using an E-R model/normalized model. data on customers. the data in the data warehouse are stored following. because of the number of tables involved. it can be difficult for users both to: join data from different sources into meaningful information and then access the information without a precise understanding of the sources of data and of the data structure of the data warehouse. the retrieval of data from the data warehouse tends to operate very quickly. to a degree.. each of the created entities is converted into separate physical tables when the database is implemented (Kimball. In the normalized approach. The dimensional approach. and salesperson responsible for receiving the order. and into dimensions such as order date. order ship-to and bill-to locations. products. The main disadvantages of the dimensional approach are: In order to maintain the integrity of facts and dimensions. because the structure is divided into measurements/facts and context/dimensions. believe in Ralph Kimball’s approach in which it is stated that the data warehouse should be modeled using a Dimensional Model/star schema.Normalized versus dimensional approach for storage of data There are two leading approaches to storing data in a data warehouse — the dimensional approach and the normalized approach. When applied in large enterprises the result is dozens of tables that are linked together by a web of joints. product number. The main advantage of this approach is that it is straightforward to add information into the database. loading the data warehouse with data from different operational systems is complicated. transaction data are partitioned into either "facts".g. In a dimensional approach. a sales transaction can be broken up into facts such as the number of products ordered and the price paid for the products. or "dimensions". etc. customer name. The normalized approach. database normalization rules. which are the reference information that gives context to the facts. whose supporters are referred to as ―Inmonites‖. A disadvantage of this approach is that. For example. Dimensional structures are easy to understand for business users. A key advantage of a dimensional approach is that the data warehouse is easier for the user to understand and to use. which creates several tables in a relational database.

Some consider it an advantage of the Kimball method. Business value can be returned as quickly as the first data marts can be created. A drill-across works by grouping (summarizing) the data along the keys of the (shared) conformed dimensions of each fact participating in the "drill across" followed by a join on the keys of these grouped (summarized) facts. the bottom-up process is the result of an initial business oriented Top-down analysis of the relevant business processes to be modelled. "Production department" resulting in a Production data mart. primarily. summarized data. that the data warehousing team has made the effort to identify . Though it is important to note that in Kimball methodology. the data warehousing effort might start in the "Sales" department." These data marts can eventually be integrated to create a comprehensive data warehouse. These approaches are not mutually exclusive. say. Ralph 2008). The actual integration of two or more data marts is then done by a process known as "Drill across". that the data warehouse ends up being "segmented" into a number of logically self contained (up to and including The Bus) and consistent data marts. and the method gives itself well to an exploratory and iterative approach to building data warehouses. The most important management task is making sure dimensions among data marts are consistent. The difference between the two models is the degree of normalization. Upon completion of the Sales-data mart. The integration of the data marts in the data warehouse is centered on the conformed dimensions (residing in "the bus") that define the possible integration "points" between data marts. Data marts contain.[3] is a proponent of an approach to data warehouse design which he describes as bottom-up. For example. and there are other approaches. Top-down versus bottom-up design methodologies Bottom-up design Ralph Kimball. The integration of data marts is managed through the implementation of what Kimball calls "a data warehouse bus architecture". by building a Sales-data mart. Facts can contain either atomic data and. Maintaining tight management over the data warehouse bus architecture is fundamental to maintaining the integrity of the data warehouse.[4] In the bottom-up approach data marts are first created to provide reporting and analytical capabilities for specific business processes. The requirement for the Sales data mart and the Production data mart to be integrable. which are dimensions that are shared (in a specific way) between facts in two or more data marts. dimensions and facts. that will be. In Kimball's words. rather than a big and often complex centralized model. a well-known author on data warehousing. if necessary.It should be noted that both normalized – and dimensional models can be represented in entityrelationship diagrams as both contain jointed relational tables. Dimensional approaches can involve normalizing data to a degree (Kimball.[5] The data warehouse bus architecture is primarily an implementation of "the bus" a collection of conformed dimensions. The single data mart often models a specific business area such as "Sales" or "Production. this means that the dimensions "conform". The business might then decide to expand the warehousing activities into the. is that they share the same "Bus".

An integration (possibly) achieved in a flexible and iterative fashion. and that the individual data marts links that information from the bus. no master plan is required upfront. Top-down design Bill Inmon. Generating new dimensional data marts against the data stored in the data warehouse is a relatively simple task. the data are static. is of critical business value.[5] Inmon is one of the leading proponents of the top-down approach to data warehouse design. the data warehouse. will not only be able to deliver the specific information that the individual data marts are designed to do.and implement the conformed dimensions in the bus. The up-front cost for implementing a data warehouse using the top-down methodology is significant. The Sales-data mart is good as it is (assuming that the bus is complete) and the production data mart can be constructed virtually independent of the sales data mart (but not independent of the Bus). If integration via the bus is achieved. which. Integrated The data warehouse contains data from most or all of an organization's operational systems and these data are made consistent. In the Inmon vision the data warehouse is at the center of the "Corporate Information Factory" (CIF). has defined a data warehouse as a centralized repository for the entire enterprise. in which the data warehouse is designed using a normalized enterprise data model. Note that this does not require 100% awareness from the onset of the data warehousing effort. are stored in the data warehouse. Non-volatile Data in the data warehouse are never over-written or deleted — once committed. Dimensional data marts containing data needed for specific business processes or specific departments are created from the data warehouse. and the duration of time from the start of project to the point that end users . and retained for future reporting. which provides a logical framework for delivering business intelligence (BI) and business management capabilities. but can deliver integrated SalesProduction information. through its two data marts. in this example either "Sales" or "Production" information. data at the lowest level of detail. that is. Time-variant The top-down design methodology generates highly consistent dimensional views of data across data marts since all data marts are loaded from the centralized repository. one of the first authors on the subject of data warehousing. read-only. often. The main disadvantage to the top-down methodology is that it represents a very large project with a very broad scope. Top-down design has also proven to be robust against business changes. Inmon states that the data warehouse is: Subject-oriented The data in the data warehouse is organized so that all the data elements relating to the same real-world event or object are linked together. "Atomic" data.

In addition. the top-down methodology can be inflexible and unresponsive to changing departmental needs during the implementation phases.[5] .experience initial benefits can be substantial.

3/ .1.98 .43909..

9.943.5574.7.80825020390/ 2-..-08 03 90 /.5574./390/.98 ..70. /.1742/110703984:7./..3-0/11.43.3/3 41 90 84:7.9.0790/ 394 805.39.3.39.394039908 . 0- 41 4398  :79072470  0./459390/203843.704:80897:.30890..80   /8.550/3 .943 394 90 /.9:7019047. 8 9.3.9./..9..39.-.98  13.7/ 94 ...70.70.088 90 31472.4.89.0839420.90880.9147.  %0 3472.5   %02.9 0307. /0700  /.9.70 74:50/ 940907 - 8:-0..90/ .0/ 897:./.0/ 9.9.5574.9..08 41 /.70 70.0  09.943 94:9 .-.// 31472.9 .3/903 .943 8 -:83088 574.8890288.0/ .9 .90 58.943. 897:.90/ 039908 8 .. #.9 7010.98... 41 90 .5  %02./2038438  .80 03..43909..90/ 94 90 47.998 897.. ..943.39039079411.439..3/8. 3 90 /.:99424/190/..5574.70 30/ 940907 - .08808 .704:80  .3/ 98/11.9.9.70 347/07942.9.9:704190/. 9. 88902 070. 43 .:8942078  574/:.9.70 89470/ 1443  94 ..9  -0.3.425.943.5574.943 7:08  %..704:80 .9.:8041903:2-07419..:9147:8078-4994 43/.0 41 98 .3.9.-083.8:702039 2-.-.-.084190/203843.04198.70.3. 570. #..9:70 /..-083.07. .80 3472..9. 1742/11070394507.8 9.3/ 4507..8 90 /2038438 8:774:3/3 902.-08 9.943.-4:99020.  90 /.3//2038438 4.9. .-08 .9.3/ 41 90 /.70 0390757808 90 708:9 8 /4038 41 9.80 :3/0789.9/408-:83088  3 90 3472.3./08/. .904708 0   /..39.9./.331:31472.704:809/.

704:8390...439.0 /.90.2.704:83 .93390/7.9.:/3 %0 :8  .2039.0  .088 8 90 708:9 41 .794190324/08. 0.798 .3/ 9070 .30/ :5 94 .9.9-0 9.0847!74/:.4314720/ /2038438  .088343.704:80 %0248925479.9.7 .98 .30.8901789/.08 /05.87.3/ 90 !74/:.08 . .4250..86:.70 349 2:9:.039:.2.70 4907 .5574.3 -0709:730/.942.9.3//203843.798 7.3.039070/4390.9 3 2-.9./0 94 05.  !74/:.990 /.9:.4314720//2038438 708/3 3 90 -:8  9.3 .79 3 90 $. -:83088 574.98  ./.9.4314720/ /2038438410.3/ 1 30.30907.9.02039 4.39.9.431472  $420 . 8 47/8  98 20.0397..90/94.79  &543 .9.792039 708:93 3 .943  %080 .5574..884190700.798  %03907.74881440/-.704:80/083.947 .39-:83088574..79 .5.708.798 8 2..-. 2094/44  90 -49942 :5 574.7488478-74:53 8:22.9. 2.3-0.9-493472.43889039/.9.0 41 90 2-.33.9.73 90/.7488  /7 . 94 /.984:/-03490/9.704:83 011479 29 89./0 7054793 .70.943 /05.943 /. 2...704:80 %0 3907. 2.70/ 3.9../09001147994/0391 .5574..850.5.70908.2.798.3/41903.9:70 8 572.28 .3 .9.70.9 /0130 90 5488-0 3907...3/ .98.089801094 ..43439008419080 74:50/ 8:22. 0 343 .943 41 .9.08 /.3/ 907.9.94341.39.9.3/ 90 . 2.2.7.7.908 394 90  8.078:8-49942 :5/0832094/4408 49942 :5/083 #.9.08  203843. !74/:.1.90/ .70 . 2.2. 801 .3.7 /2038438.5 2-.9:70 ( %0 /.39 94 3490 9.798  %0 ...70.79.4390084190 8.9.4250943 41 90 $. .. 3:2-07 41 4. 2.439.798 .3390390794190/. .704:80 -:8 .943 54398 -09003 /..8 $.9434194472470/.439...9.43889039  3 2-.3 4390/ 70.9..-08  %0 /110703.3-07057080390/30399 70.943 %080/..9.39...8 -49 .-908 147 850..03472.9.438/07 9 ..2.0/974:90 2502039./.94341/.088.70/ 1. .990/.08808  %4: 9 8 25479.79  %0 706:702039 147 90 $. 2.-0  8 9. 94 2..2.882.-03907. .70//.70/ . . %0830/.3/ 3.90/ 94 574.:947 43 /.9.9.94385 /.0/24/0 :83088.3.943.9 2-. . .82.7.3.9..2.9.8 . -:83088 470390/ %45 /43 .704:80 -:8 .0 .439. 2.90. -090031.90.7988903/430-.33 99 2.3 572.3.990 8.79 94 -0 3907.9.08.70.-:83088..3. /.90.3 2502039.5   %45 /43.5574. .1.943 /.94.3.5574.020399.9434190/. 574543039 41 .5574.9.9:70 8 1:3/.20:8 9.392. . .79 %0 -:83088 29903 /0.8-49942 :5 ( 3 90 -49942 :5 .704:83 ( 8 .42570038.3/.7.704:808. /.8:.9079. #.0880894-024/00/  .70 /20384389..38 9.. ..9.704:80 03/8 :5 -03 8020390/ 394 .850.3/1.704:80 -:8 . 2..1.08 /..  .243 /.0 -09003 90 94 24/08890/0700413472..9..:8.704:808  47 0.250  90 /..70 1789 .9.3/ . 9. 94 -:/3 /.40.7 8:22.9 90 /2038438. $.798390/.3/902094/.:0.943 41 90 -:8 .798.9.3907.9.3 39..792039  - -:/3 ./0700 2-.. 2094/ 9.3 8:70 /2038438 ..574.24/08.0/08..3 0547.98394472470/.1.07 90 /.0/ .08 /.2.4..9.95.9.7-08.9.798.

0 9490945 /432094/44 89.5574.9 90 4089 0.79 .9.943  .4.703088174290438094190 /. /.9.9.889028. .1.0880847850.7030/940907  43 .704:83  . .489 147 25020393 .705740.0/ 039075780 /.0/ 70548947 147 90 03970 039075780 ( 3243 8 430 41 90 0.89470/390/.-094/0.70 30.2.9.9.8 /0130/ ./0/174290.79 41 5740.9..79844/.9./3 57454303984190945 /43.9.9.704:80  974: 98 94 /. 2.798.9..073-:8308839003.3/9.4314720//2038438390-:8 .9. 2.94/.3/ 9080/.0 .79 -:93493/0503/0394190:8  1 3907.39  . 3 90 /..93 30 /203843.9 94 90 5439 9. 3472...041/09. 90 -:8 8 ..798. 47/0.704:80 8 47.70.9..704:80 8 .943 84507. /.90/ $.9./:.9./ 43 . .30/1471:9:707054793  3907.70/0830/94/4  3 98 0. .2.9 41 /.9.798 .0/70548947 %45 /43/083.2..9439.3 /0.908.9./05.9903/.43889039/203843.39 %0945 /43/0832094/440307.308  0307..9.3/-:83088 2.082509.9.7089470/390/..943 ./:.990/.03907 41 90 47547.43889039  %20 .99.9.347.8 %02.03 94 -0 74-:89 .0397.3 -0 . 24/0  942. 70... 3/0503/03941908.0.9./.9 90 .7.798   349 43-0. .943./.. .0841/. .9. -:83088 ..9.3.704:80.  9.3/709.70.943 41 920 1742 90 89.5.943 5488-  .20 70.07 3907.3/ 90 /:7.79.33/.0/3.79:.9.9.1.425090  ./.2.01.9.-908  324389.88:23 9..3/ 90 574/:.9.0  ./..02039. /.943 31472.9.9. .38/. .70.1.08 47 !74/:. 0020398 70.70 .9.9089.9.9 470390/ %0 /.39...997057080398.9 .943 ./8.17.07 -74..08 /.704:80 .943  -:9 ..1742248947.9.843  %45 /43/083  3243  430 41 90 1789 ./08.3/8.93 94 90 8.039474-0. 2..9.798389...31472.389 -:83088 .9.2.90/174290/.450  %0 :5 17439 .3/250203990.4. -:83088574.:0  3 3907.9.9..8 .9.08 !74/:.90/ .8 . 3 90 /.90 31472.0/  90 /.07 . 90 /.2047147/0.943 /.0.90/.:9478 43 90 8:-0.843 90 /.704:808 /0830/ :83 ..422990/  90 /.89075.2.43897:.704:80 .704:80 :83 90 945 /43 2094/44 8 831.9 03/ :8078  43.2...9 90 -:8 8 .  41903  8 41 ..79883..9 31472..9.07 4.0397.08/./.3/907...704:808.947  .38990/.41./0.704:83011479 342.943 31472. /.7920398.704:808 $:-0.439.3.9.0790850.704:80/083 3. .9.250 0907 $.07 79903 47 /0090/ 89.439.300/0/ 147850.574.9 8  /.798.943174290-:8 4909.9..90 .702. .30/ 84 9.84 574.704.9.9903/.704:80 3 90 3243 .7488 /.704:80 203843.10-0.90/ %0/.998/408349706:70 .898 .0.38706:70/:517439 %0$.

.808 (  .3 -0 310-0. -030198 .  3 .9435.39.300/8/:73902502039.//943  90 945 /43 2094/44 ..3/:37085438.050703.33/05.094.792039.0 39..3 -0 8:-89.