Data Warehouse Material Concepts

ABC Datawarehouse Modeling and Design
Unit 1. An Introduction to Data Warehousing
What is Data warehousing?

According to Bill Inmon, known as the father of Data Warehousing, a data warehouse is a
subject oriented, integrated, time-variant, nonvolatile collection of data in support of
management decisions.
• Subject-oriented means that all relevant data about a subject is gathered and stored as
a single set in a useful format;
• Integrated refers to data being stored in a globally accepted fashion with consistent
naming conventions, measurements, encoding structures, and physical attributes, even
when the underlying operational systems store the data differently;
• Non-volatile means the data warehouse is read-only: data is loaded into the data
warehouse and accessed there;
• Time-variant data. The relevance of time-variant is in the sense of data getting added
on as time goes on. Time being the most important dimension, etc.
Data warehousing is a concept. It is a set of hardware and software components that can be used
to better analyze the massive amounts of data that companies are accumulating to make better
business decisions. Data Warehousing is not just data in the data warehouse, but also the
architecture and tools to collect, query, analyze and present information.
Unit 1.1 Data warehousing concepts
Operational vs. informational data:
Operational data is the data you use to run your business. This data is what is typically stored,
retrieved, and updated by your Online Transactional Processing (OLTP) system. An OLTP
system may be, for example, a reservations system, an accounting application, or an order entry
application.
Informational data is created from the wealth of operational data that exists in your business and
some external data useful to analyze your business. Informational data is what makes up a data
warehouse. Informational data is typically:
• Summarized operational data
• De-normalized and replicated data
• Infrequently updated from the operational systems
• Optimized for decision support applications
• Possibly "read only" (no updates allowed)
• Stored on separate systems to lessen impact on operational systems
ABC Ver 1.10 Page 1

A data warehouse is a “subject-oriented, integrated, non-volatile, time variant collection of data

in support of management decisions [Inm].” The end-users of a data warehouse are usually
business analysts, as distinct from field personnel or call takers.
Question: What do you think is the skill profile of the data warehouse end-user?
Operational Decision support

Data Content Current values Archival, summarized, calculated data
Data Organization Application by application Subject areas across enterprise
Nature of Data Dynamic Static until refreshed
Data Structure & Complex; suitable for operational Simple; suitable for business analysis
Format computation
Access Probability High Moderate to low
Data Update Updated on a field-by-field basis Accessed and manipulated; no direct
update
Usage Highly structured repetitive Highly unstructured analytical
processing processing
Response Time Sub-second to 2-3 seconds Seconds to minutes
Source: [STG]
Question: Do the descriptions under “Data structure & format” fit in with the skill profiles of the
respective end-users?
A data mart is a scaled down deployment of a data warehouse that contains data focusing on a
departmental user’s analytical requirements. For example, the Ohio-based Huntington Bank
Corporation set up a data mart for its general ledger system, to get the ledger system's functional
information to the bank's financial analysts and budget coordinators quickly.
Data mining is the process of examining data for trends and patterns that might have evaded
human analysis. For example, Shoko’s Sunday circulars contained coupons advertising health
and beauty aids, consumables, and household chemicals, which were are all located on the left-
hand side of the stores. Shoko’s data mining exercise revealed that people who were coming in
to shop gravitated to the left-hand side of the store for the promotional items and were not
necessarily shopping the whole store. Consequently, it added apparel promotions to the Sunday
circulars.
An on-line Analytical Processing (OLAP) application is intended to provide end-users an ability
to perform any business logic and statistical analysis that is relevant. This analysis must happen
fast, i.e., it must deliver most responses to users within about five seconds, with the simplest
analyses taking no more than one second and very few taking more than 20 seconds.
Multidimensional databases are non-relational DBMS products that are specialized for use for
the kinds of queries in data warehouses. This is in contrast to using specialized analysis tools
that run on top of a traditional RDBMS.
What is the ROI for a data warehouse? A recent study [Fis] of 45 major companies by the
International Data Corporation found an average three-year return on investment in data
warehouse systems of 401%. More instructive is the very wide range of returns reported by the
companies, from 16,000 percent to minus 1,857 percent. Moral: data warehousing is not a silver
bullet; use with care!
Multi-dimensional data structures can be implemented with multidimensional databases or
extended RDBMSs. Relational databases can support this structure through specific database
designs (schema), such as "star-schema", intended for multi-dimensional analysis and highly
indexed or summarized designs. These structures are sometimes referred to as relational OLAP
(ROLAP)-based structures.
ABC Ver 1.10 Page 2

Metadata/Information Catalogue:
Metadata describes the data that is contained in the data warehouse (e.g. Data elements and
business-oriented description) as well as the source of that data and the transformations or
derivations that may have been performed to create the data element.
Unit 1.2 Benefits of Data Warehousing
A well designed and implemented data warehouse can be used to:

• Understand business trends and make better forecasting decisions
• Bring better products to market in a more timely manner
• Analyze daily sales information and make quick decisions that can significantly affect
your company's performance
Data warehousing can be a key differentiator in many different industries. At present, some of
the most popular Data warehouse applications include:
• sales and marketing analysis across all industries
• inventory turn and product tracking in manufacturing
• category management, vendor analysis, and marketing program effectiveness analysis
in retail
• profitable lane or driver risk analysis in transportation
Unit 1.3 Datawarehousing Application Class: How it has been

evolved.
Throughout the history of systems development, the primary emphasis had been given to the
operational systems and the data they process. But there is a difference in the fundamental
requirements of the operational and analysis systems are different: the operational systems need
performance, whereas the analysis systems need flexibility and broad scope. It has rarely been
acceptable to have business analysis interfere with and degrade performance of the operational
systems.
Data warehousing has quickly evolved into a unique and popular business application class.
Early builders of data warehouses already consider their systems to be key components of their
IT strategy and architecture.
In building a Datawarehouse application the source inputs are listed below.
1. Data from legacy systems.
In the 1970’s virtually all business system development was done on the IBM mainframe
computers using tools such as Cobol, CICS, IMS, DB2, etc. The 1980’s brought in the new
mini-computer platforms such as AS/400 and VAX/VMS. The late eighties and early nineties
made UNIX a popular server platform with the introduction of client/server architecture. By
some estimates, more than 70 percent of business data for large corporations still resides in the
mainframe environment.
2. Extracted data from micro desktop databases.
In recent times advanced users will frequently use desktop database programs that allow them to
store and work with the information extracted from the legacy sources. Many desktop reporting
and analysis tools are increasingly targeted towards end users and have gained considerable
popularity on the desktop.
ABC Ver 1.10 Page 3

Another fall side to this is the difficulty in sharing analyses with others, e.g. during budgeting,
one user (say the boss) may create analysis models (say allocation rules) that are to be used by
all others. The first user then generates the final output by putting these analyses together.
Furthermore, semantics of the data may need to be standardized for use before letting it out to
the users. In a desktop environment, this may be nearly impossible.
As the data is stored on disparate systems, it is very difficult to ensure that updates to the data
are communicated to all users, e.g. say sales data comes in, and one person sends brand-wise
summaries to some key users who then forwards them to his sub-ordinates. Some hours after that
it is realized that data from one of the warehouses was missed out, and revised reports are sent.
Result: Different people working on different versions of the same data. Unnecessary
reconciliation issues crop up later.
3. Decision-Support and Management Information System ( MIS)
The last category of analysis systems has been decision support systems and executive
information systems. Decision support systems tend to focus more on detail and are targeted
towards lower to mid-level managers. Executive information systems have generally provided a
higher level of consolidation and a multi-dimensional view of the data, as high level executives
need more the ability to slice and dice the same data than to drill down to review the data detail.
This category is somehow close to Datawarehousing applications, but it has the following defect.
• These systems have data in descriptive standard business terms, rather than in cryptic
computer fields names. Non-technical user design data names and data structures in these
systems for use.
Datawarehousing Applications are in prominence today because there are key technology is
available, hardware prices are down, good Server software, availability of internet
applications, Most importantly lots of tools are available too.
Unit 1.4 Self Review through a Case study.
A large Grocery chain, which has large stores around 500 in 3 states. This has many departments
also. Each store deals with around 60000 products. 40000 products are brought from external
vendors. Rest 20000 is prepared from different departments.
Each product has a product code.
1. Which is the primary key?
2. What is the Place for Data Collection?
3. What are the different business activities?
4. On what the management will be interested in
5. What should be Business Goal?
6. What is the Grain?
7. What is the business measurement s for the Fact table?
8. Give an approximate Database size. Size of the Fact table.
ABC Ver 1.10 Page 4

After answering the above questions give an attempt to following conceptual questions.
1. Define grain statement.
2. Define measure.
3. Find difference between OLTP and OLAP. Supply two SQL’s to justify both system
4. Justify how the importance of Time with respect to both OLTP and OLAP.
5. Is OLAP and Datawarehousing go together.:
ABC Ver 1.10 Page 5

ABC Data Warehouses: Modeling and Design
The dimensional model

Traditional normalized database designs are inappropriate for data warehouses for 2 reasons:
• DSS processing can involve accessing hundreds of thousands of rows at a time across
several tables. Complex joins can seriously compromise performance
• Usage/access paths in an OLTP environment are known a priori. In DSS, usage is very
unstructured; users often decide what data to analyze moments before they request it, and
applications cannot be hardcoded for a particular schema
Data modeling is a useful design tool because it allows automatic generation of normalized
database schema from an ER diagram. Because traditional normalized database designs are
inappropriate, traditional data modeling is also inappropriate as a design tool. It continues to be
a useful tool for modeling and understanding business information (the business essence) in a
technology-independent way, and provides a foundation for mapping the data in the operational
data stores to that in the data warehouse.
Database designs for data warehouses
follow a star schema. There are one Course Student
or more central fact tables, each
“surrounded” by several dimension
tables that provide the foreign keys
that define each fact table row. The Attendance Fact
fact table is like a transaction table
while the dimension tables are like Course_id
master tables. For example, if the Student_id
attendance of a student in a course is Date_id
a “transaction,” the associated Instructor_id
dimensions would be Course, #of_days_attended
Instructor, Student and Course_date. lab_days_attended
For each such “transaction”, we might
like to store no_of_days_attended;
this would then be the fact of the
transaction. Note that Date_id is
Instructor Time
actually a foreign key. Having a
separate table called time is essential for performing the kind of temporal analysis that is
typically required in a business context.
The driver of data warehouse design is the nature of the standard data warehouse query, which is
“Give me [aggregated] facts broken down by dimensions D1 and D2 for such-and-such time
period.” For example, we might be interested in looking at attendance by Course by Day, or the
sum of attendances by Course by Year, by Student by Quarter, and so on. This translates into
SQL which looks something like this:
select D1.attrib_1, D2.attrib2, sum(F.fact1), sum(F.fact2)
from fact F, dimension1 D1, dimension2 D2, time T
where F.dim1_key = D1.key
and F.dim2_key = D2.key
and F.timekey = T.key
and T.quarter = ‘1Q1997’
group by D1.attrib1, D2.attrib2
order by D1.attrib1, D2.attrib2
Exercise: Write an SQL that reveals the number of attendances from SBU1 by Course by Year.
Note that time is conceptually just another dimension. A dimensional constraint or filter such as
T.quarter = ‘1Q1997’ is called an application constraint.
Exercise: You run a grocery store chain. Your sales fact table records, for each sale, the UPC,
the date and time, store id, details of promotional offers on the product sold, and of course the
ABC Ver 1.10 Page 6

sale quantity and selling price. Imagine the kinds of temporal analyses you would like to do with
your sales data. What attributes would you like the Time dimension table to have? How large
could this dimension table possibly get?
Browsing is the activity of exploring a single-dimension table, prior to firing the template query
above, with a two-fold purpose:
(i) choosing attributes for the select clause of the query; this might be done by simply
dragging the attribute name onto a graphic representing the template query or report.
(ii) choosing application constraints for selecting a subset of rows of the table for the query.
Consider the completely denormalized product table, with each row containing
information about product, product category, and package description. The browsing
activity might result in the application constraint Package_desc = “TetraPak” after a
select package_desc where category = “beverage”.
Drilling down is the action of dragging an attribute name (from a dimension table) onto an
existing report.
The size of a dimension table is invariably a tiny fraction of the size of the fact table. Besides, a
data warehouse is updated only once a day, and it is only a small fraction of these days on which
a dimension table is ever updated. The dimension tables are thus left completely unnormalized.
A policy of having completely unnormalized dimension table allows graphical browsing and
automatic SQL generation for all standard user queries of the kind mentioned above.
It also pays to have as many descriptive or qualifying attributes for each dimension as can be
imagined, so that the end-user can set a variety of application constraints. (Consider, for
example, an analyst who wants to know how the sale of paints on Holi (the festival) days differs
from sales on other days.)
There is a subtle difference between a dimension
and an entity. “Time” is a dimension, but is Day Month Quarter Year
associated with a great many entities, as shown in
the accompanying figure. For convenience, one of
the hierarchies (the one most commonly used in Week 4-week
queries) is usually designated the primary period
dimension, and every other hierarchy a secondary
dimension. Each level in a hierarchy is said to
roll-up to the next level (though it is apparent that 13-week
roll-ups are not always uniquely defined). period
Dimensional modeling, then, attempts to depict
the facts and their associated dimensions, without explicitly depicting the entities and
relationships that make up a dimensional hierarchy.
Exercise: You want to analyze course attendances as well as course nominations. Develop a star
schema to do this. Would you choose to have one fact table or multiple? If you choose the
former, will you have any new dimensions?
In general, a single fact table is a good idea where multiple types of facts share a subtype-
supertype relationship with the bulk of the attributes being common. For example, transactions
involving bank accounts have different flavors depending on whether it is a savings account or a
checking account that is being operated. This difference in flavor manifests itself as mild
variations in the composition of attributes that make up the transaction fact. With relatively
unrelated fact types on the other hand, the number of common attributes is small, so the
preferred choice is to have a custom fact table for each fact type, and replicate common
attributes in all custom fact tables to avoid joins.
Exercise: What happens to a table that represents a dimension which has subtypes?
It is common to have data points (facts) that are described as an adjective of your base data (e.g.
actual sales and budgeted sales). Rather than anticipate all adjectives during warehouse design,
we can create a partitioning dimension that holds only the adjectives and their descriptions
(each adjective is called a partition), with the fact table row containing a column called just
ABC Ver 1.10 Page 7

sales, along with a new foreign key called partition. This makes it easy to add a new type of
fact such as “forecast sales” : just insert a row in the partition for the new adjective “forecast”,
and have the fact table foreign key partition indicate “forecast” for each record that represents a
forecast.
Unit 1.5 Issues in Dimensional Modeling
Big dimensions
Denormalization increases redundancy and, consequently, size. Sometimes a fully denormalized
dimension table does become uncomfortably large. When this happens, the dimension may be
normalized or snowflaked in the following manner. The dimension table stores one key for each
level of the dimension's hierarchy. The lowest level key joins the dimension table to the central
fact table. The rest of the keys join the dimension table to the corresponding higher-level tables.
In a snowflake schema, every dimension is normalized in this manner. The word “snowflake”
refers to the shape of the fully normalized schema when represented graphically.
Exercise: Snowflake the Time dimension. How will you handle multiple hierarchies? Do you
think Time is a big dimension?
There is significant difference of opinion about whether snowflaking should be done at all even
for big dimensions. Ralph Kimball [Kim] believes it should never be done, while the Stanford
Technology Group (an Informix company) believes it is useful, since in a dimension table of
500,000 rows, it is conceivable to save two megabytes per row through normalization and hence
save a full gigabyte of disk. Kimball compares the saving to the overall size of the typical
warehouse which is about 50 GB. Performance is also a factor to be considered: without
snowflaking, a query that needs to analyze sales by brand will have to rummage through 500,000
product rows to filter out perhaps a score brands. Another factor is the complexity of the
structures as perceived by the end-user (a business analyst), and the associated loss of browsing
ability (recall the definition of browsing given earlier). Finally, load programs and overall
maintenance become more difficult to manage as the data model becomes more complex.
A dimension such as Customer may have many qualifying attributes, such as age, sex, and
income_level, which are of interest to the business analyst not as specific values but as a
combination of brackets. Instead of retaining these as individual attributes in the customer
dimension table, we can replace these by a demographics_key that points to a row in a
minidimension table as shown:
Demographics Sales fact Customer dimension

minidimension
time_key customer_key
demographics_key customer_key demographics_key
age_bracket demographics_key first_name
income_bracket product_key address
sex .... ....
....
Such a schema speeds up queries with complex demographic conditions. Also note that not all
combinations need to be stored in the minidimension table—a customer in the age group <10 is
unlikely to be in a high-income category.
Question: Why make demographics_key a part of the fact table?
Dimensions change their characteristics, albeit slowly. When a customer changes her address,
we can either modify the address in the customer’s record in the customer dimension table (and
lose historical information) or insert a new customer dimension record to capture history.
ABC Ver 1.10 Page 8

Exercise: You may be able to limit the number of changes of interest; for example, you might
rarely analyze data that is over a year old, and it is rare for marital status to change more than
three times a year. How could you handle this efficiently?
A demographics_key can also undergo changes over time. A customer may move into a higher
income bracket, requiring a change in demographics_key. It is easy to see that
demographics_key can be treated just like an attribute, albeit a more complex one.
Unit 1.6 Aggregation strategies

Data-intensive queries access a large number of rows. Data-selective queries touch only a few
rows, but contain complex and diverse selection criteria. Data warehouse end-users take more
strategic decisions and hence execute a large number of data-intensive queries, unlike OLTP
end-users. A typical data-intensive query would be “give me sales by region for each brand.”
Pre-aggregation strategies are required to reduce response time for such queries. A simple
measure of the need for pre-aggregation is to compute the compression ratio, which is (the
number of rows reported)/(the number of rows retrieved). Full aggregation refers to the
precomputation and storage of all possible aggregates (i.e., combinations of all levels of all
dimensions).
Exercise: How would you estimate the increase in the overall database size for full aggregation?
How many additional tables are necessary to store precomputed aggregates? The level
technique allows the answer to this question to be zero. With this technique, every aggregate
fact is stored in the base fact table. Fact table rows that stored product category sales by store by
day would have the product_key point to rows in the Product table that identified a product
category rather than an individual product. A level field in the Product table would have level =
“category” for such rows and level = “base” for rows representing individual products.
Exercise: How would you determine the total number of fact and dimension tables for full
aggregation using a different dimension table for each level and a different fact table for each
aggregate?
The level technique and separate table technique are of course two ends of the [full aggregation]
spectrum. With a hybrid approach, some aggregates can be stored using the level technique
while others can be stored in separate tables. It may be better to use the level technique for cases
where new facts keep pouring in.
The alternative to precomputation is dynamic or SQL-based aggregation, which is meaningful
for aggregates that are not computed often enough to warrant precomputation. For example,
aggregate sales for product categories can be computed by selecting SUM(sale_value) and
grouping by category. Aggregates for the complete product hierarchy (sales by sub-category,
category, brand, etc.) can be computed by successive select statements that group at the
respective level.
Again, a hybrid approach is possible along this axis: for example, monthly totals may be
precomputed (preaggregated) while yearly totals may be arrived at using SQL-based aggregation
of monthly totals.
Aggregate navigators are tools that allow user applications to fire base-level SQL as though they
were performing pure dynamic aggregation. The navigator maintains definitions of the current
aggregation table structure, and uses this to rephrase the SQL to access the relevant aggregate
tables instead. The algorithm typically looks for the smallest aggregate fact table whose
associated dimension tables contain all the dimensional attributes required for the supplied
query. Once this is done, the base-level fact and dimension table names are replaced with the
aggregate fact and dimension table names.
select category_description, sum (qty) select category_description, sum (qty)
from sales_fact, product, store, time from category_sales_fact, category_product, store,
where time
{ join conditions on product, store & where
time } { join conditions on category_product, store & time }
and store.city = Cincinnati and store.city = Cincinnati
and
ABC time.day = 01011996 and 1.10
Ver time.day = 01011996 Page 9
group by category_description group by category_description
Exercise: Can you use partitioning dimensions to handle adjectives for aggregates? How about
“Average” itself as an adjective?
Unit 1.7 Self review Case study on Dimensional Modeling
The following case study is to be read and observed with the solution given At the end of it one
must be very clear how to draw a Dimensional model.
The key to creating an efficient MDDB application is thorough analysis of both the data and its
users. After the data elements have been identified for reporting to the end users, the business
entities will fall in distinct groups of variables with similar characteristics or dimensions.
For example, consider a sales organization, which sells articles to different customers through
different suppliers spread at various geographical locations. From the transaction data (base
tables) of the organization, we can design a fact table which contains the denormalized sales
data at a granularity which is required for creating an MDDB. This fact tables stores the Units
and the Dollars of the sales volume for at a daily level.
The fact table for this case can be outlined as follows:
Product Geography Supplier Time Units Dollars
In this example, the first 4 columns represent the key determinants of the two facts (Products
sold in Units and Dollars).
In an MDDB model, the fields of the four Dimensions must intersect to determine the values of
the facts. To create the dimensions for the MDDB which is to be built from the base table, it
advisable to have dimension tables for each of the dimensions. These dimension tables should be
used to derive the dimension fields, hierarchies and other attributes, if required.
In this case, the dimensions could be derived as follows:
PROD GRP PROD FAMILY ARTICLE
G1 F1 A1
G1 F2 A2
ABC Ver 1.10 Page 10

Product
Geography COUNTRY REGION SHOP

C1 R1 S1
C2 R2 S2
Supplier GROUP SUPP SUPPLIER

G1 SU1
G2 SU2
YEAR QTR MONTH DAY

Time
Y1 Q1 M1 D1
Y1 Q1 M1 D2
Measure
Measure Code Measure Name Precision
Units Products Sold in Units Unit
Dollars Products Sold in Dollars K
Note that
 This design complies with the classic STAR schema design of a
Datawarehouse
 The fact table contains a compound primary key, with one segment for each
dimension, and additional columns of additive, numeric facts
 Each dimension in the design has a defined hierarchy. The parent-child
relationship (additive/semi-additive/non-additive) could be business driven.
Alternatively, the dimension tables can be designed to have the attribute level
indicator of each record
 Each dimension contains (and not restricted to) a key segment
 Deriving a dimension from a table in an MDDB is always advisable because of
the following primary reasons:
 Dimension size can be reduced by selecting only the valid dimension
fields from the table. Thus, reducing the size of the MDDB
 Modifications to the dimension hierarchies can be handled easily
 Facilitates better maintenance of the cube build process
 Facilitates standardization of dimensions across different MDDB
applications in an organization
 Descriptions and levels of the dimension fields can be stored in the
tables
 Facts of the data have been clubbed into a Measure dimension to store
different attributes of the facts and handle any changes in future (for example,
Precision is one of the properties of the facts included here).

Unit 2. Data Warehouse Architecture
Unit 2.1 Data Warehousing Architecture Model
The following components should be considered for a successful implementation of a Data

Warehousing solution: [an]
• Open Data Warehousing architecture with common interfaces for product integration
• Data Modeling with ability to model star-schema and multi-dimensionality
• Extraction and Transformation/propagation tools to load the data warehouse
• Data warehouse database server
• Analysis/end-user tools: OLAP/multidimensional analysis, Report and query
• Tools to manage information about the warehouse (Metadata)
• Tools to manage the Data Warehouse environment
Transforming operational data into informational data:

Creating the informational data, that is, the data warehouse, from the operational systems is a
key part of the overall data warehousing solution. Building the informational database is done
with the use of transformation or propagation tools. These tools not only move the data from
multiple operational systems, but often manipulate the data into a more appropriate format for
the warehouse. This could mean:
• The creation of new fields that are derived from existing operational data
• Summarizing data to the most appropriate level needed for analysis
• Denormalizing the data for performance purposes
• Cleansing of the data to ensure that integrity is preserved.

Even with the use of automated tools, however, the time and costs required for data conversion
are often significant. Bill Inmon has estimated 80% of the time required to build a data
warehouse is typically consumed in the conversion process.
Data warehouse database servers--the heart of the warehouse:

Once ready, data is loaded into a relational database management system (RDBMS) which acts
as the data warehouse. Some of the requirements of database servers for data warehousing
include: Performance, Capacity, Scalability, Open interfaces, Multiple-data structures, optimizer
to support for star-schema, and Bitmapped indexing . Some of the popular data stores for data
warehousing are relational databases like Oracle, DB2, Informix or specialized Data Warehouse
databases like RedBrick, SAS.
To provide the level of performance needed for a data warehouse, an RDBMS should provide
capabilities for parallel processing - Symmetric Multiprocessor (SMP) or Massively Parallel
Processor (MPP) machines, near-linear scalability, data partitioning, and system administration.
Data Warehousing Solutions - what is hot?
Solution Area Product Vendor

Report and Query Impromptu Cognos
BrioQuery Brio Technology
Business Objects Business Objects Inc
Crystel Reports Seagate Software
OLAP / MD analysis DSS Agent/Server Microstrategy
DecisionSuite Information Advantage
EssBase Hyperion Solutions
Express Server Oracle Corp.
PowerPlay Cognos Corporation
Brio Enterprise Brio Technology
Business Objects Business Objects
Data mining Enterprise Miner SAS Institute
Clementine SPSS
Discovery Server Pilot Software
Intelligent Minor IBM
Darwin Thinking Machines
Data Modeling ER/Win Platinum
Data extraction, transformation, load DataPropagator IBM
InfoPump Platinum Technology
Integrity Data Re-Eng. Vality Technology
Warehouse Manager Prism Solutions
PowerMart Informatica
Databases for data warehousing DB2 IBM
Oracle Server Oracle
MS SQL Server Microsoft
RedBrick Warehouse Red Brick Corp.
SAS System SAS Institute
Teradata DBS NCR
• profitability analysis or risk assessment in banking

• claims analysis or fraud detection in insurance
Unit 3. Issues in Datawarehousing Projects.

It is important to recognize the issues involved in building and hence in managing the
Datawarehouse. Interestingly some user may say that a data warehouse that is only 50 gigabytes
is not a full-fledged data warehouse, and they may refer to it instead as a data mart. For a
smaller company, 50 gigabytes or even much less can represent every relevant piece of
information covering last 10 years and can well represent a powerful data warehouse.

The issues a project leader must keep in mind.[sas]

1. For separating the data for business analysis from the operational data.
2. The logical transformation of the data, including data warehouse modeling and de-
normalization of the data
3. The issues associated with physical transformation of the data.
4. The generation of summary views.
Unit 3.1 Carrying Data from OLTP to Warehousing data
These issue is here how to separate and when to separate because the operational data from
analysis data have not significantly changed with the evolution of the data warehousing systems,
except that now they are considered more formally during the data warehouse building process.
In the analysis and design phase building Datawarehouse is done through a journey from
existing ER model. Advances in technology to producing standard reports, today’s data
warehousing systems support very sophisticated online analysis including multi-dimensional
analysis.
Data warehousing systems are most successful when data can be combined from more than one
operational system. When the data needs to be brought together from more than one source
application, it is natural that this integration be done at a place independent of the source
applications. The primary reason for combining data from multiple source applications is the
ability to cross-reference data from these applications. Nearly all data in a typical data
warehouse is built around the time dimension.
The data warehouse system can serve not only as an effective platform to merge data from
multiple current applications; it can also integrate multiple versions of the same application. For
example, an organization may have migrated to a new standard business application that
replaces an old mainframe-based, custom-developed legacy application. The data warehouse
system can serve as a very powerful and much needed platform to combine the data from the old
and the new applications. Designed properly, the data warehouse can allow for year-on-year
analysis even though the base operational application has changed.
Operational systems are designed for acceptable performance for pre-defined transactions. For
example, an order processing system might specify the number of active order takers and the
average number of orders for each operational hour. Even the query and reporting transactions
against the operational system are most likely to be predefined with predictable volume.
Even though many of the queries and reports that are run against a data warehouse are
predefined, it is nearly impossible to accurately predict the activity against a data warehouse.
Data is mostly non-volatile. This attribute of the data warehouse has many very important
implications for the kind of data that is brought to the data warehouse and the timing of the data
transfer.
Many data warehousing projects have failed miserably when they attempted to synchronize
volatile data between the operational and data warehousing systems.
In short, the separation of operational data from the analysis data is the most fundamental data-
warehousing concept. Not only is the data stored in a structured manner outside the operational
system, businesses today are allocating considerable resources to build data warehouses at the
same time that the operational applications are deployed.
Unit 3.2 Issue 2 :Logical transformation of operational data

The data is logically transformed when it is brought to the data warehouse from the operational
systems. The issues associated with the logical transformation of data brought from the
operational systems to the data warehouse may require considerable analysis and design effort.
The architecture of the data warehouse and the data warehouse model greatly impact the success
of the project. This section reviews some of the most fundamental concepts of relational

database theory that do not fully apply to data warehousing systems. Even though most data
warehouses are deployed on relational database platforms, some basic relational principles are
knowingly modified when developing the logical and physical model of the data warehouses.
Importance of the possibility of synchronized data in the source systems, e.g. if the product
codes are not standard across the source systems, and product attributes are stored across
systems, it becomes impossible to maintain all the product attributes in the warehouse. This is
one of the most important concerns to be taken care of before initiating a data-warehousing
project. While data scrubbing and cleaning can take care of the past data, for continuous updates
in an efficient manner, these requirements become essential.
The data warehouse model needs to be extensible and structured such that the data from
different applications can be added as a business case can be made for the data. A data
warehouse project in most cases cannot include data from all possible applications right from the
start. Many of the successful data warehousing projects have taken an incremental approach to
adding data from the operational systems and aligning it with the existing data. Data warehouse
model aligns with the business structure
A data warehouse logical model aligns with the business structure rather than the data model of
any particular application. The same logic can be applied to entities in an entity relationship
diagram, which are used as the starting point for operational systems. Though the relevant points
are being covered – i.e. narrow definition of entities in applications and the need to create one
consolidated attribute base – the impact is not felt – perhaps because a direct comparison with
ER modeling is not made. I feel we should also introduce the concept of an enterprise data
model here.
Unit 3.2.1 De-normalization of data

A data modeler in an operational system would take normalized logical data model and convert
it into a physical data model that is significantly de-normalized. De-normalization reduces the
need for database table joins in the queries.
Some of the reasons for de-normalizing the data warehouse model are the same as they would be
for an operational system, namely, performance and simplicity. Static relationships in historical
data. Another reason that de-normalization is an important process in data warehousing
modeling is that the relationship between many attributes does not change in this historical data.
Another important example can be the price of a product. The prices in an operational system
may change constantly. Some of these price changes may be carried to the data warehouse with
a periodic snapshot of the product price table. In a data warehousing system you would carry
the list price of the product when the order is placed with each order regardless of the selling
price for this order . maintain dynamic relationships between business entities, whereas a data
warehouse system captures relationships between business entities at a given time.
Unit 3.3 Issue 3 : Physical transformation of operational data

Historical data and the current operational application data are likely to have some missing or
invalid values Physical transformation of data homogenizes and purifies the data. These data
warehousing processes are typically known as “data scrubbing” or “data staging” processes.
Physical transformation includes the use of easy-to-understand standard business terms, and
standard values for the data. A complete dictionary associated with the data warehouse can be a
very useful tool. During these physical transformation processes the data is sometimes “staged”
before it is entered into the data warehouse. The data may be combined from multiple
applications during this “staging” step or the integrity of the data may be checked during this
process.

The terms and names used in the operational systems are transformed into uniform standard
business terms by the data warehouse transformation processes. It is important to give single
physical definition of an attribute.As an attribute is defined physically for the data warehouse, it
is essential to use meaningful data types and lengths. Use the standard data length and data type
for each attribute everywhere it is used. A functional data dictionary can facilitate this
consistent use of physical attributes. Second important point is to use consistently entity attribute
values
All attributes in the data warehouse need to be consistent in the use of predefined values.
Different source applications invariably use different attribute values to represent the same
meaning. These different values need to be converted into a single, most sensible value as the
data is loaded into the data warehouse. Or, if the data is to be used by the same set of users, one
may need to store the different attributes too, so that users do not see a disconnect between their
operational and decision support systems.
A far more important problem is inconsistent definition and use of the entities themselves, e.g.
some applications may be storing information at the price code detail level (encoded in the
product code), while others may be storing at a planning code level (all price codes, variants,
etc. are clubbed). Moreover, because of user habits, some of the codes used in the planning
system may be outdated, and replaced by new codes in the sales system. Clubbing information
from multiple sources then becomes a big problem.
Unit 3.4 Issue 4: Data with default and missing values to be

interpreted consistently.
The data brought into the data warehouse is sometimes incomplete or contains values that
cannot be transformed properly. It is very important for the data warehouse transformation
process to use intelligent default values for the missing or corrupt data. It is also important to
devise a mechanism for users of the data warehouse to be aware of these default values.
Some data attributes can easily be defaulted to a reasonable value when the original is missing
or corrupt. Other values can be obtained by referencing other current data. For example, a
missing product attribute such as unit-of-measure on an order entity can be obtained by
accessing the current product database. Some attributes cannot be filled by defaults for missing
values. In fact, it may be dangerous to attempt to assign default for certain types of missing
values. A poor default may corrupt the data and lead to invalid analysis at a later stage. In these
cases, it is safest to leave the missing values as blank. In some cases, it may make sense to pick
a specific value or symbol that indicates a missing value.
February is not stored in the data warehouse. Also, missing data for part of the year prevents
any meaningful year-on-year analysis.
It is important to design a good system to log and identify data that is missing from the data
warehouse. When a user runs a query against the data warehouse, it is essential to understand
the population against which the query is run.
Accurate and complete transformations help maintain the integrity of the data warehouse.
Unit 3.5 Issue 5: Mapping Data to reflect Business view

Summary views often are generated not only by summarizing the detail data but also by
applying business rules to the detail data. For example, the summary views may contain a filter
that applies the exact business rules for considering an order a sale or a filter that applies the
business rules for allocating a sale to a channel entity. The summary views can hide the
complexities of the detail data from the end user for many, if not most, analysis tasks.
The business rules that are applied in generating summary views can be complex. These
business rules may determine exactly what constitutes a sale or they may determine how a sale
is allocated to a sales or channel entity. In addition to applying the business rules while
generating summary views, the data warehousing system may perform complex database

operations such as multi-table joins. Product sales may be computed by joining the Sales,
Invoice, and Product tables. The criteria to join these tables may be complex. While
individuals mining data in the warehouse detail records need to understand all the complexities
of business rules, most users can retrieve effective summary business information without fully
understanding the detail data.
The single most important reason for building the summary views is the significant performance
gains they facilitate. The summary views in a data warehouse provide multiple views into the
same detail data. These views are predefined dimensions into the detail data. These views
provide an efficient method for the analyst to link with the detail data when necessary.
Unit 3.6 Issue 6: Selecting Tools to be used against the data

warehouse
In most data warehousing projects, there is a need to select a preferred data warehouse access
tool for the most active users. A small number of users generate most of the analysis activity
against the data warehouse. The data warehouse performance can be tuned to the requirements
of the tool appropriate for these active users. This tool can be used for training and
demonstration of the data warehouse. A user can start with a low-level tool that is already
familiar to him or her. After becoming familiar with the data warehouse he or she may be able to
justify the cost and effort involved with using a more complex tool.
Unit 3.6.1 An Evaluation Checklist
The choice of an OLAP tool for a particular environment and application depends on
the key requirements of the analysts, programmers and the end-users of the
application. Before getting into the details of subjecting an OLAP tool to any
evaluation criteria and performing any test, one needs to have the performance
requirements very clear to guide the evaluation process. Some of the focus areas
can be found out by having the following questions answered at the outset:[das/rak]
Data Access Features

What would be the final data format?
What are the common selection criteria to be used?
Are mathematical operations (addition, subtraction etc.) critical to
the selection ?
Are statistical operations (Statistical functions -mean, average,
standard deviation etc.) critical?
Use of other data manipulation features Sort, Discard, and Filter,
Shifting critical?
Data Exploration Features

Is graphical representation very important?
Are traffic light analysis, key performance analysis, pattern
matching, and lifetime analysis useful for the users?
Usability
How much of OLAP familiarity exists with the users?
Do the users have any quantified performance expectations?
Are expert options, user programming required?
Is GUI a deciding factor?
What is the tolerable online response time?
Size and Scalability

How big is the is the current volume of data?
What is the data growth potential in future?

How fast is the data growing?

Are the users platform-specific for the tool?
What is the batch update window?
What is the maximum number of dimension for a single MDDB?
What is the maximum aggregation level?
Critical Resources
What are the critical System resources that need to be optimally
used by the OLAP tool?
What are the resources that are factors for evaluation?
Set of Features
What features will be high priority for the users?
Will the features help the users do the work more productively?
Limitations and Constraints
What are the limitations and constraints that should be ruled out?
Are there any problems in the current tool(if any) ?
Unit 3.7 Performance considerations

Physical design for a data warehouse is concerned primarily with query performance and less
with storage or update performance. Queries can be speeded up in two ways: by speeding up the
retrieval of rows from an individual table, and by speeding up the multiple-table join process.
Speeding up retrieval. A bitmap index creates an array where the columns are the domain of
the indexed field and the rows correspond to the rows of the table. If we indexed Marital_Status
with values Single, Married and Other, we would have three columns in the bitmap. Each value
in the array is an on/off bit that indicates the value of the field in the corresponding row. This
indexing scheme speeds up row selection by the use of bitwise operations.
Bitmap indexes are often used in conjunction with B-trees. Suppose you want to index
product_category for which there are 1000 distinct values. Suppose there are 100 leaves for the
B-tree; each leaf will then represent a range of 10 values for the field being indexed
(product_category). Then a bitmap can be maintained at each leaf; each such bitmap will have
the same number of rows as before but only 10 columns. Bitmap-based techniques are suitable
only for low-cardinality data (i.e., the indexed field must have no more than a couple of hundred
distinct values). For medium and high cardinality data, a more traditional B-tree implementation
is usually used.
Not surprisingly, bitmap-based indexing schemes are costly to update, but this is okay:
remember that updates are a rare phenomenon in a data warehouse!
The use of aggregates can relieve the pressure to build indexes. A query that does not constrain
over a given dimensional level (e.g., “get sales by brand by region does not constrain on the
product dimension levels below the level “brand”) can be redirected to a suitable aggregate
table. Thus only one sort order on the master composite index on the fact table needs to be built.
In the sales data warehouse for example, this composite index could be time by product by store.
Put differently, only queries that constrain on the lowest levels will use the base fact table and
this composite index.
Speeding up joins. Traditional databases typically join two tables at a time; this can be
disastrous for data warehouse queries. Worse, the performance varies dramatically with the
order in which the tables are joined. A DBMS specialized for data warehouses will instead
proceed as follows. First all dimensional constraints are evaluated and a list of respective
primary keys is generated. These are combined to generate a sorted list of composite keys that is
matched with the fact table index which is itself a sorted list of composite keys. Note that this
process can be parallelized, and indeed is, by leading database vendors.

Another possibility is to reduce the number of dimensional tables by creating dimensions whose
instances are actually combinations of two or more dimensions. If, for example, we were
interested only in 200 brand-region combinations, a brand-region dimension table containing
200 rows could replace the individual brand and region tables.
Storage: aggregate explosion. Full aggregation is often dangerous in practice. Consider a sales
data warehouse where there are 10000 products and 100 stores. On any given day, not all
products are sold in all stores; perhaps only 1% of the possible 10 6 combinations actually occur.
Yet, most products will be sold somewhere, and each store will sell something, so that both
product-wise and store-wise [daily] aggregates are relevant. This number is itself 10100, so that
the database size will double if these aggregates are precomputed. Adding a dimension (say
customer) will clearly compound the problem: (Exercise: how?) The term-compounded growth
factor (CGF) [Pen] indicates the database size with full aggregation as a multiple of its size with
no pre-aggregation, and is usually between 1.5 and 2.5 per dimension.
The solution to this problem is trial and error: use business analysis and query patterns to decide
what aggregates are worth precomputing. Pareto’s law can be expected to hold in this case: 80%
of queries will utilize only 20% of all possible aggregates, so that it is possible to meet
performance requirements adequately by constructing only a fraction of all possible aggregates.
Unit 3.8 Risks in Datawarehousing Projects.
* Definitions of data are always inconsistent across user types, upstream data systems and
applications. No 2 users/systems agree to a common definition easily. This will happen in
RA/Design Stage.
* Data ownership in data warehouses is very less. So the sanctity of the data is most of
the times suspected. This comes out as a problem only when an application is developed to show
the data to the users. This will happen in Testing/Implementation stage.
* High dependency on Up-stream systems. Any delays in making the up-stream
interfaces ready affect the RA/Design/Development cycle. This will happen in
RA/Design/Development.
* In a reporting application, problems mostly originated from the up-stream systems are
attributed to the application. This gives rise to end-user dissatisfaction. This will happen in
Acceptance/Testing/Implementation.
* Obtaining test data for data validation is a risk if real time data is of very high
confidentiality. This will happen in Development/Testing stage.
* Squeezed development cycle due to high visibility of the reporting application. the
background process of collecting data from different sources is not visible to the users. This will
happen in Development/Implementation stage.
* Data-processing time needs to be minimized to ensure availability of most up-to-date
data worldwide at all times. This will happen in Production/Implementation stage.
* The datamart /warehouse support team is located in a country, it is often expected to
address issues from users located in different time zones. This will happen in post-
production/maintenance stage.[das]
Unit 3.9 Case study: Insurance

This case study will touch upon almost all concepts that have been introduced before.
Think of an insurance company that insures automobiles, homes and individuals. A transaction
is related either to policy formulation or to claims processing. An insurance company sells
coverages, and the data warehouse is to be used to assess the profitability of the existing
coverages. Also of importance is the efficiency of claims redressal.
Exercise: List some queries that would be appropriate for an insurance data warehouse to
address.

Because a transaction is related either to policy formulation or to claims processing, we can have
two fact tables, one for policy formulation and one for claims processing.
Question: What about having just one fact table? Conversely, what about having one fact table
for each type of claim (small/large) or each type of policy (domestic/automobile/industrial)?
The policy creation and claims processing fact tables have the following structures (attributes in
italics indicate those which will also appear in monthly policy and claims snapshots.
Policy Creation Fact Table

Claims Processing Fact Table
TransactionDate: time dimension with alternate
hierarchies TransactionDate:
EffectiveDate: SYNONYM of TransactionDate EffectiveDate:
InsuredParty#: big, dirty dimension InsuredParty#
Employee# (Agent/Broker/Rater/Underwriter) Employee#: the authorizer of the claim
Coverage#: This is the company’s “product.” May be Coverage#
represented nicely as a subtype/supertype CoveredItem#
hierarchy. Policy#
CoveredItem#: Each Coverage specifies some Claimant#: usually a dirty dimension
Covereditems. Claim#: a codified description of a claim
Policy#: Quite possibly a degenerate dimension ThirdParty#: (Witness/Expert/Payee). This assumes
TransactionType: (Create/Rate/Underwrite/Cancel...) only one predefined third party is involved.
Fact: Set of transaction attributes functionally TransactionType: (Open/Set reserve/Inspect/Pay...)
determined by the above. Attribute set may Fact: Set of transaction attributes functionally
differ for each coverage. determined by the above
Additional policy creation snapshot attributes: Additional claims processing snapshot attributes:
Premium accrued OutstandingClaims: number of outstanding claims
Premium due at a point in time (semi additive).
No-claims bonus (derived)
Questions:
• InsuredParty is a dirty dimension, which means that multiple instances of InsuredParty
may actually represent the same insured party. How can we clean it?
• Does CoveredItem need to be distinguished from Coverage? Or does a coverage
automatically specify a CoveredItem too? Is CoveredItem a big dimension? Will
CoveredItem figure in a fact table representing a monthly policy status snapshot?
• The Rater may assign a risk_grade to the policy during the Rate transaction. To which
table/dimension does risk_grade belong?
• A degenerate dimension is one which has no separate dimension table. How might Policy
be a degenerate dimension?
• The CoveredItem called “automobile” has attributes different from the CoveredItem called
“computer.” Are Automobile and Computer subclasses (subtypes) or instances of
CoveredItem?
• When and how would you split the policy fact table to handle subtypes represented by
different attribute sets of Fact? Would this require queries to always access more than one
fact table?
• Should the attributes of a claim (as represented by the codified identifier Claim#) constitute
a dimension or should they be part of the FACTS of the claims processing fact table? Note
that the attribute set of a claim depends on the coverage scheme.
Coverages come in a huge number of flavors, just as do customers (insured parties). The set of
attributes which are specified most often as a combination to browse coverages can be hived off
into a minidimension. Examples of such attributes could be risk_level and market_segment.
The subset of rows in the policy creation fact table that represent policy cancellations may have
no useful FACT attributes, since the purpose of each such row is merely to record the fact of
cancellation of an existing policy. If we fragment the fact table horizontally so as to create a
separate table to store just policy cancellations, we have what is called a factless fact table. Each

row of a factless fact table contains only foreign keys corresponding to the relevant dimensions,
but no fact attributes per se.
A factless fact table can also be used to store information about coverages and covered items
that did not have any buyers in, say, a given month. In the simplest case this table would have
columns for the foreign keys Coverage#, CoveredItem# and Month. At the end of each month,
a row would be inserted into this table for each Coverage/Covered item combination that did not
attract any buyers (did not figure in any new policy). This is an example of a coverage table (not
to be confused with insurance coverages).
Exercise: Write a SQL to determine the number of coverage/covered item combinations that did
not figure in any new policy in a given month.
Aggregates. Imagine a requirement to report, for each coverage, the total premiums received
and total claim payments made by calendar month. This needs to be further aggregated to report
total premium and claims payments made by month. Note that the requirement to “aggregate
across policies” usually implies a requirement to also aggregate across other dimensions such as
Employee and InsuredParty to get a useful result.
Exercises:
• How would you design the warehouse to handle a query along the lines of “How many of
our customers have chosen money-back policies?” [Hint: this may require some changes to
the operational system too.]
• If each precomputed aggregate is stored in a separate aggregate fact table, what is the
maximum number of such aggregate fact tables required?
Workout: A sales and marketing data warehouse needs to be built for a manufacturer of hospital
health care products. Salesmen are assigned territories, which roll up to districts, regions and
areas. A product rolls up to subgroup, group and family, where a subgroup is defined by the
assembly line that it rolls out from. (A single manufacturing facility may, of course, have more
than one assembly line.) The products are typically bought by hospitals through buying groups
to which the respective hospitals belong, though a hospital may sometimes buy through a direct
contract or even through a buying group in which it is not a member. We need to be able to
analyze these different types of sales. Design the data warehouse.
Unit 3.10 Self-Review: A Case Study to arrive at a complete

Datawarehouse Solution.
Data Warehousing for Finance Systems
About this case study…
Customer Name A leading computer manufacturer

Industry Computer Hardware and Software Manufacture
Project(s) Name Data Warehousing
Service offered RA/Design/Development/Implementation/Warranty
Focus Area Data Reporting for Finance and Operations
Technology RDBMS/OLAP
User Profile Finance Managers, Analysts, Planners and CFO
office
Application Features

 24*7 Worldwide Access

 Executive Level Slicing/Dicing/Drilling
 Global application with regional granularity
 Seamless DSS to corp. changes
 2 Data warehouses, 1 Datamart and 6 OLAP Cubes
 4 Years Historical, Current and 2 years Forecast data available for analysis
on MDDB
 Upstream/downstream interfaces
 Ad-hoc and Canned reporting
 Bookmarking
 Business KPIs
 Robust security features
 Admin features
 Bulletin Board
Client Profile:
The client designs, manufactures and markets personal computers and related
personal computing and communicating solutions for sale primarily to
education, creative, consumer and business customers. It leads the area of
computing in revolutionary products and innovative designs in all aspects.
Client's driver for the warehousing project:
Why did the customer have to undertake this project?

What were his drivers? What did he want at the end of the day?
The OLAP Reporting Systems started as part of an initiative for meeting

the analytical reporting needs of Client's Finance & Operations executives
through presentation of useful data on a timely and accurate manner.
Finance executives at Client's site needed key global and regional Actual,
Plan and Forecast data for Trend Analysis, Visualization, Budgeting,
Planning, and Modeling to support Decision-Making. The transaction
systems did not segment or aggregate data for business analysis so, there
was a need to have a common, consistent, fast and analysis-ready tool for
the Finance users world-wide.
The financial data being too sensitive, a two layer user security was
required to prevent users from accessing data outside their area as well as
during freeze period.
Before this project, Finance data available on Mainframe systems was

manually processed to produce some custom reports for top-management.
The access was restricted and limited to user expertise for data within a
certain timeframe.
Unit 4. OLAP FUNDAMENTALS.

In 1993, E.F. Codd & Associates published a white paper, commissioned by Arbor Software
(now Hyperion Solutions), entitled 'Providing OLAP (On-line Analytical Processing) to User-
Analysts: An IT Mandate'. Dr Codd is, of course, very well known as a respected database
researcher from the 1960s through to the late 1980s and is credited with being the inventor of the
relational database model, but his OLAP rules proved to be controversial due to being vendor-
sponsored, rather than mathematically based. [olap]
Basic Features
1. Multidimensional Conceptual View Dr Codd, believe this to be the central core of OLAP.
2. Intuitive Data Manipulation. Dr Codd prefers data manipulation to be done through direct
actions on cells in the view, without recourse to menus or multiple actions.
3 Accessibility:. In this rule, Dr Codd essentially describes OLAP engines as middleware, sitting
between heterogeneous data sources and an OLAP front-end. Most products can achieve this, but
often with more data staging and batching than vendors like to admit.
4 Batch Extraction vs Interpretive. This rule effectively requires that products offer both their
own staging database for OLAP data as well as offering live access to external data. Today, this
would be regarded as the definition of a hybrid OLAP, which is indeed becoming the most
popular architecture, so Dr Codd has proved to be very perceptive in this area.
5: OLAP Analysis Models. Dr Codd requires that OLAP products should support all four
analysis models that he describes in his white paper (Categorical, Exegetical, Contemplative and
Formulaic). Perhaps Dr Codd was anticipating data mining in this rule?

6: Client Server Architecture .Dr Codd requires not only that the product should be
client/server but that the server component of an OLAP product should be sufficiently intelligent
that various clients can be attached with minimum effort and programming for integration. This
is a much tougher test than simple client/server, and relatively few products qualify. Perhaps he
was anticipating a widely accepted API standard, which OLE DB for OLAP is expected to
become.
7: Transparency. This test is also a tough but valid one. Full compliance means that a user of,
say, a spreadsheet should be able to get full value from an OLAP engine and not even be aware
of where the data ultimately comes from. Like the previous feature, this is a tough test for
openness.
8: Multi-User Support .Dr Codd recognizes that OLAP applications are not all read-only and
says that, to be regarded as strategic, OLAP tools must provide concurrent access (retrieval and
update), integrity and security.
Special Features
9: Treatment of Non-Normalized Data. This refers to the integration between an OLAP engine
and de-normalized source data. Dr Codd points out that any data updates performed in the
OLAP environment should not be allowed to alter stored de-normalized data in feeder systems.
regarded as calculated cells within the OLAP database.
10: Storing OLAP Results: Keeping Them Separate from Source Data. This is really an
implementation rather than a product issue. In effect, Dr Codd is endorsing the widely held view
that read-write OLAP applications should not be implemented directly on live transaction data,
and OLAP data changes should be kept distinct from transaction data..
11: Extraction of Missing Values. All missing values are cast in the uniform representation
defined by the Relational Model
12: Treatment of Missing Values. All missing values to be ignored by the OLAP analyzer
regardless of their source.
Reporting Features
13: Flexible Reporting. Dr Codd requires that the dimensions can be laid out in any way that
the user requires in reports. We would agree, and most products are capable of this in their
formal report writers. Dr Codd does not explicitly state whether he expects the same flexibility
in the interactive viewers.
14: Uniform Reporting Performance. Dr Codd requires that reporting performance be not
significantly degraded by increasing the number of dimensions or database size. Curiously,
nowhere does he mention that the performance must be fast, merely that it be consistent. There
are differences between products, but the principal factor that affects performance is the degree
to which the calculations are performed in advance and where live calculations are done (client,
multidimensional server engine or RDBMS). This is far more important than database size,
number of dimensions or report complexity.
15: Automatic Adjustment of Physical Level. Dr Codd requires that the OLAP system adjusts
its physical schema automatically to adapt to the type of model, data volumes and sparsity.
Dimension Control
16: Generic Dimensionality. Dr Codd takes the purist view that each dimension must be
equivalent in both its structure and operational capabilities. However, he does allow additional
operational capabilities to be granted to selected dimensions (presumably including time), but he
insists that such additional functions should be grantable to any dimension.
17: Unlimited Dimensions & Aggregation Levels. Technically, no product can possibly
comply with this feature, because there is no such thing as an unlimited entity on a limited
computer. In any case, few applications need more than about eight or ten dimensions, and few
hierarchies have more than about six consolidation levels..
18: Unrestricted Cross-dimensional Operations .Dr Codd asserts, and we agree, that all forms
of calculation must be allowed across all dimensions, not just the 'measures' dimension. In fact,

many products that use only relational storage are weak in this area. Most products with a
multidimensional database are strong. These types of calculations are important if you are doing
complex calculations, not just cross tabulations, and are particularly relevant in applications that
analyse profitability.
In OLAP server data stored in three different ways .
• Multidimensional OLAP (MOLAP)

• Relational OLAP (ROLAP)
• Hybrid OLAP (HOLAP)
MOLAP
MOLAP is a high performance, multidimensional data storage format. With MOLAP, data is
stored on the OLAP server. MOLAP gives the best query performance, because it is specifically
optimized for multidimensional data queries. MOLAP storage is appropriate for small to
medium-sized data sets where copying all of the data to the multidimensional format would not
require significant loading time or utilize large amounts of disk space.
ROLAP
With ROLAP data remains in the original relational tables. A separate set of relational tables is
used to store and reference aggregation data. ROLAP is ideal for large databases or legacy data
that is infrequently queried.
HOLAP
HOLAP combines elements from MOLAP and ROLAP. HOLAP keeps the original data in
relational tables but stores aggregations in a multidimensional format. HOLAP provides
connectivity to large data sets in relational tables while taking advantage of the faster
performance of the multidimensional aggregation storage.

References and bibliography:

[Fis] L. Fisher (1996) “Along the Infobahn: Data Warehouses” in Strategy & Business, Booz, Allen and
Hamilton, Inc.
[Guff] F. McGuff (1997). “Data Modeling for Data Warehouses,” http://members.com/fmcguff/dwmodel
[Inm] W. H. Inmon. (199?) Building the Data Warehouse, John Wiley, NY.
[In2] W. H. Inmon. (1996) Creating the Data Warehouse Data Model from the Corporate Data Model,
Prism Solutions Tech Topic Vol.1 No. 2., Prism Solutions Inc., Sunnyvale, CA.
[Kim] R. Kimball. (1996) The Data Warehouse Toolkit, John Wiley, NY. Primary reference
[Mer] M. E. Meredith and A. Khader (1997), “Divide and Aggregate: Designing Large Warehouses,”
technical report, Miller Freeman Inc.
[STG] “Designing the Data Warehouse on Relational Databases,” technical report, Stanford Technology
Group, Inc (an Informix Company).
[Pen] N. Pennies (1997) “Database explosion,” Business Intelligence Ltd.
Data Warehousing - For Better Business Decisions
[an]Anjaneyulu Marempudi (marempudi@msn.com)
[dash] Data Warehousing For Finance Systems by Assis Dash , Infosys
http://www.sas.com [sas] SAS Institute website
[olap] What is OLAP by N.Pendese by Business Intelligence Limited 2000
[dash/rakesh] Multidimensional database tool evaluation. By Assiss Das and Dr. Rakesh Agarwal.

Data Warehouse Material Concepts

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Warehouse Material Concepts

Uploaded by

Copyright:

Available Formats

ABC Datawarehouse Modeling and Design

Unit 1. An Introduction to Data Warehousing

What is Data warehousing?

Unit 1.1 Data warehousing concepts

Operational vs. informational data:

ABC Ver 1.10 Page 1

A data warehouse is a “subject-oriented, integrated, non-volatile, time variant collection of data

Operational Decision support

ABC Ver 1.10 Page 2

Unit 1.2 Benefits of Data Warehousing

A well designed and implemented data warehouse can be used to:

Unit 1.3 Datawarehousing Application Class: How it has been

In building a Datawarehouse application the source inputs are listed below.

1. Data from legacy systems.

2. Extracted data from micro desktop databases.

ABC Ver 1.10 Page 3

3. Decision-Support and Management Information System ( MIS)

Unit 1.4 Self Review through a Case study.

Each product has a product code.

1. Which is the primary key?

2. What is the Place for Data Collection?

3. What are the different business activities?

4. On what the management will be interested in

5. What should be Business Goal?

6. What is the Grain?

7. What is the business measurement s for the Fact table?

8. Give an approximate Database size. Size of the Fact table.

ABC Ver 1.10 Page 4

1. Define grain statement.

5. Is OLAP and Datawarehousing go together.:

ABC Ver 1.10 Page 5

The dimensional model

ABC Ver 1.10 Page 6

ABC Ver 1.10 Page 7

Unit 1.5 Issues in Dimensional Modeling

Demographics Sales fact Customer dimension

ABC Ver 1.10 Page 8

Unit 1.6 Aggregation strategies

Unit 1.7 Self review Case study on Dimensional Modeling

Product Geography Supplier Time Units Dollars

In this case, the dimensions could be derived as follows:

PROD GRP PROD FAMILY ARTICLE

ABC Ver 1.10 Page 10

Geography COUNTRY REGION SHOP

Supplier GROUP SUPP SUPPLIER

YEAR QTR MONTH DAY

ABC Ver 1.10 Page 11

Unit 2. Data Warehouse Architecture

Unit 2.1 Data Warehousing Architecture Model

The following components should be considered for a successful implementation of a Data

Transforming operational data into informational data:

ABC Ver 1.10 Page 12

Data warehouse database servers--the heart of the warehouse:

Data Warehousing Solutions - what is hot?

Solution Area Product Vendor

• profitability analysis or risk assessment in banking

Unit 3. Issues in Datawarehousing Projects.

ABC Ver 1.10 Page 13

The issues a project leader must keep in mind.[sas]

Unit 3.1 Carrying Data from OLTP to Warehousing data

Unit 3.2 Issue 2 :Logical transformation of operational data

ABC Ver 1.10 Page 14

Unit 3.2.1 De-normalization of data

Unit 3.3 Issue 3 : Physical transformation of operational data