WHITE P APER A New Data Warehouse Appliance and Reference Architectures: HP and Microsoft Introduce Integrated Technology Options

for Mission-Critical Requirements

Sponsored by: HP Dan Vesset December 2010
www.idc.com

IDC OPINION
At a time of great economic volatility, managers and executives of most large organizations have experienced a significant decrease in the ability to rely effectively on experience or intuition to make decisions. They are finding that old cause-andeffect mental models are becoming less relevant, while the demand to respond faster and with greater insight to ongoing internal and external events based on facts is increasing. As a result, more large organizations, in both commercial and public sector industries, are turning to business intelligence (BI) and data warehousing collectively referred to by IDC as business analytics solutions. Organizations that have adopted these solutions to enable better decision making report positive results. IDC research shows that managers at the most competitive organizations are two times more reliant on business analytics (rather than intuition) than managers at their least competitive peers. The most competitive organizations in an industry indicate two times more frequently that output of BI solutions is very influential on actions taken by their employees. As evidence of the value of BI solutions to drive competitive differentiation mounts, leading organizations have recognized the need to take full advantage of all the available relevant information to raise the level of BI of their entire workforce. However, many organizations are constrained by the current realities of having to do more with less. IT budget and staff cuts in the recent past are exacerbated by an environment of growing data volumes and growing numbers of business end users requesting access to information to improve their own decision-making capabilities. Many are now looking for more prepackaged BI solutions that enable rapid deployment and optimal query performance, as well as scalability, availability, and security features appropriate to these increasingly mission-critical decision support and decision management systems.

Global Headquarters: 5 Speen Street Framingham, MA 01701 USA

P.508.872.8200

F.508.935.4015

THIS IDC WHITE P APER EXAMINES

In this white paper, IDC highlights the latest research about demand for and benefits of integrated data warehousing solutions. Data warehousing solutions such as an integrated hardware and software appliance as well as predefined reference configurations are discussed as two key options for deployment. The paper introduces joint HP and Microsoft data warehousing solutions and provides recommendations for large organizations evaluating data warehousing solutions for the purpose of supporting improved decision-making processes.

SITUATION OVERVIEW
Business Intelligence Needs on the Rise
In the beginning of 2010, despite the weak economy or perhaps because of it, large organizations (as measured based on the number of employees) expected to increase their spending on business analytics software to 5% of their total software spend. In 2009, the last full calendar year, the total business analytics software market grew in low single digits despite the overall IT market's decline of close to 5%. In IDC's October 2010 BI and analytics survey, 56% of respondents from large organizations indicated that they are planning to increase the number of BI solution users in the next 12 months (another 38% are planning to keep the number of users unchanged). In the same survey, 65% of respondents from large organizations indicated that they are planning to increase the number of BI projects in the next 12 months. These are just a few indicators suggesting growing demand for BI and data warehousing technology as awareness of the potential benefits of better BI has increased over the past few years. For example, IDC research has uncovered quantitative evidence suggesting a link between greater analytical orientation of an organization and its competitiveness. In a study published in 2009, IDC evaluated over 1,100 organizations across 11 countries and identified key indicators of pervasive BI and four BI competency levels for each of these indicators. As part of this research we segmented organizations into leaders (most competitive organizations in an industry) and laggards (least competitive organizations). IDC analysis showed that 80% of leaders but only 58% of laggards have the highest level of analytical orientation a set of policies, procedures, and processes enabled by BI solutions. More publicity about successful BI solution deployments and their impact, as well as articles and books in business press such as The Wall Street Journal, Harvard Business Review, and MIT Sloan Management Review, has elevated the subject of BI and analytics to the executive suite at many more large organizations. More executives are wondering if their organizations are missing out on opportunities to make more informed decisions based on historical trend analysis and predictive analytics. The pressure on IT to deliver BI support to end users is mounting, and there are a number of different technologies for providing such support, including the data warehouse platform, which forms the foundation for most successful projects tasked with improving support for end-user decision making.

Data Warehouse Platform and Options

IDC's data warehouse platform definition includes software for data warehouse generation and management as well as the accompanying server and storage hardware. The technology options range from standalone software, server, and storage components to reference architectures based on pretested hardware and software configurations to fully integrated hardware and software appliances. In a 2009 IDC research project investigating the challenges of BI and data warehousing projects, 39% of about 1,000 large organizations indicated that the design and deployment time of BI and data warehousing projects is too lengthy, 46% indicated that

#225855

2010 IDC

the requirement to have special skill sets for implementation is a challenge, 46% indicated that complex data preparation and data model development is a challenge, and 54% indicated that high cost is a challenge. Based on the challenges these organizations reported to IDC, we believe that preconfigured and preintegrated solutions exhibit many characteristics that can benefit large organizations.

Benefits of Integrated Data Warehouse Solutions

There are several benefits of integrated data warehouse solutions that in combination highlight the mission-critical nature of the solutions to a growing number of large organizations. For example, when reporting on the expected and realized benefits of a solution such as a data warehouse appliance, respondents to an IDC survey indicated that better query performance is the top reason for deploying the appliance, followed by faster time to deployment and better support and customer service. Simpler acquisition. Prepackaged or integrated solutions can take several forms. Most frequent options include appliances or reference configurations. In both cases, the technology vendor or vendors test software and hardware components to come up with recommended configurations given a certain level of data volume, number of users, and typical workloads on the proposed data warehouse. By identifying their own specific requirements, technology buyers can then select the most appropriate configuration as a starting point for their BI solution. By substantially decreasing the need to decide which individual technology component should be included in a purchase, vendors simplify the technology acquisition process for the end-user organization. Scalability. Current market trends require data warehouses to scale to support terabytes of data and hundreds, and sometimes thousands, of concurrent users. Forty percent of large organizations indicated that their largest data warehouse is at least 10TB, and over half plan to increase the number of users with access to the BI solutions. In addition, a data warehouse strategy needs to take into account that successful data warehousing solutions will need to grow and evolve over time as user requirements, amount of data, and number of users change. IDC research indicates that data warehousing strategies vary by company size. For example, 25% of large organizations have a central enterprise data warehouse as their primary data warehousing strategy, 33% prefer multiple independent data marts, and 51% prefer some combination of both approaches. However, regardless of the deployment strategy, all data warehousing projects are incremental. Therefore, a technology solution should be able to scale over time as end-user requirements increase. This may require a scale-up strategy of deploying increasingly powerful server technology or a scale-out strategy that utilizes massively parallel processing (MPP) to string together large numbers of relatively smaller systems. Purpose-built, integrated solutions provide end users with an appropriate choice of product depending on their specific needs at a point in time while enabling incremental growth of the system.

2010 IDC

#225855

Optimized performance. Optimized performance is another benefit of integrated solutions. Given a set of data and user characteristics, technology vendors optimize the solution to address the most prevalent query workloads. Although much of the discussion about data warehouse performance centers on query performance as experienced by end users, it is just as important to mention the performance of the data integration process. This process includes steps for data extraction, transformation, and loading into the data warehouse. The need to consolidate data from multiple sources and update data warehouses at both batch and real-time intervals within shrinking time windows is an important indicator of more pervasive BI and analytics. Satisfaction with the data update frequency is one of the six key indicators of pervasive BI and analytics and a distinguishing factor between the most competitive organizations and the least competitive organizations. Availability. Data warehouses and the BI solutions they support are no longer "nice-to-have" technology used by a few quantitative analysts. As already mentioned, market research points to the growing criticality of data warehouses and the BI solutions they support as a key factor in providing competitive differentiation enabled by better decision making. In today's "intelligent economy," a growing number of data warehouses are becoming mission critical as they are deployed to support ongoing operational and tactical decision-making processes, not just the less frequent strategic decisions. In IDC's October 2010 BI and analytics survey, 17% of respondents from large organizations indicated that their BI solution can't be out of service (unavailable) for more than one hour before such an outage will have material negative impact on the business (almost a third of respondents said the BI solutions can't be down for more than six hours). As pretested and optimized solutions, integrated data warehouse technology that combines the appropriate hardware and software configurations can draw on the expertise of data warehousing specialists to provide end users with highly available solutions that meet their requirements. Support costs. For all of the above mentioned reasons, preintegrated solutions can have lower ongoing support costs. When users trust their vendors' recommendations and deploy the technology as suggested (even if with some modification), vendors are in a better position to respond to any support request faster and at a lower cost to both the user and to their own organizations. In addition, using commonly available software and hardware skills can help keep down internal technology support costs. Entering the preintegrated data warehousing solution market, HP and Microsoft have jointly engineered an enterprise-class data warehouse appliance. This represents a significant change in how Microsoft customers can deploy their mission-critical data management capabilities. The data warehouse appliance enables faster implementation, optimized performance, higher availability, and peace of mind with joint support from HP and Microsoft. In addition to the appliance strategy, HP and Microsoft are offering a broad range of solutions for data warehousing technology. For example, HP and Microsoft provide a full set of reference architectures for data warehousing that extend from the medium-sized organization to the large enterprise as well as custom implementation services.

#225855

2010 IDC

Business Intelligence Solution from HP and Microsoft

HP and Microsoft have had a longstanding partnership for delivering data warehousing solutions. Most recently, the companies have solidified this relationship with further investments in technology, staff, and support services for bringing data warehousing solutions to large organizations that look to integrated or prepackaged solutions that simplify or eliminate some of the steps in data warehouse technology acquisition, development, deployment, and support processes. The data warehousing solutions can be purchased and deployed either as an appliance or as reference architectures.

Data Warehouse Appliance

The jointly engineered HP and Microsoft data warehouse appliance presents prospective users with an integrated, pretested technology solution. The core technology in the data warehouse appliance includes Microsoft SQL Server 2008 R2 Parallel Data Warehouse a technology that bundles data integration, data warehousing, reporting, and analysis functionality in a single software product. The close partnership between HP and Microsoft provides large organizations with this software preconfigured on an appliance platform based on HP's market-leading ProLiant servers and StorageWorks storage technology. Customers now can purchase a highly scalable, optimized appliance for their most demanding workloads. The HP Enterprise Data Warehouse Appliance optimized for Microsoft SQL Server 2008 R2 Parallel Data Warehouse is available for organizations seeking the benefits of the fully integrated, optimized hardware/software solution for projects with a requirement for faster implementation and scale from 37TB to over 500TB.

Reference Data Warehouse Architecture

For customers seeking a more customized solution, the two vendors offer a set of data warehousing reference architecture configurations. HP solutions optimized for Microsoft SQL Server Fast Track Data Warehouse come in a range of pretested configurations that support data warehouses with less than 1TB and up to 48TB. The Fast Track Data Warehouse reference configurations are based on prescriptive guidelines developed by the two vendors and are optimized for query workloads patterned for large sequential reads.

Different But Complementary: Data Warehouse Appliance and Reference Architectures

When evaluating data warehousing solutions, technology decision makers need to understand immediate as well as potential future business requirements. They need to assess a given technology solution against all the criteria discussed in the earlier section on the benefits of integrated data warehouse solutions. Inevitably there are tradeoffs among different technology options that need to be considered. For example, the HP Enterprise Data Warehouse Appliance optimized for Microsoft SQL Server 2008 R2 Parallel Data Warehouse would be the more appropriate solution for a mission-critical data warehouse or a high-scale targeted data mart for specialty analytic workloads. The Fast Track Data Warehouse Reference Architecture may be

2010 IDC

#225855

more appropriate for lower-scale data warehouses and/or data marts or in situations where a customer may want to have flexibility or purchase software and hardware components separately. In addition, the two offerings from HP and Microsoft are acquired and deployed in different ways. The appliance comes with the hardware preassembled and with the software installed and configured to the extent possible outside of a customer's environment. Reference architectures enable customers to purchase a Fast Track Data Warehouse Reference Architecture that is, a hardware bundle from HP and separate SQL Server software from Microsoft and build the data warehouse based on the reference configuration on their own or with the help of a data warehousing systems integrator. With the HP Enterprise Data Warehouse Appliance and Fast Track Data Warehouse, the range of solution choices gives large organizations the option to start and/or grow their data warehousing solution incrementally as business requirements dictate. Additionally, the investments in Fast Track Data Warehouse solutions can be maximized by redeploying them as dependent data marts or "spokes," receiving data from the HP Enterprise Data Warehouse Appliance or "hub" in a hub-and-spoke model.

Beyond Software and Hardware: Data Warehousing Services

The technology its functionality, performance, and cost is not the only factor that should be considered by prospective HP and Microsoft customers. IDC research shows that after-sales support ranks as one of the top factors that technology buyers consider in their data warehousing solution selection process. To that end, HP and Microsoft provide customers with joint support services for the HP Enterprise Data Warehouse Appliance. Customers choose from various levels of support service from both HP and Microsoft to meet their business needs. With Microsoft as the first point of contact for support, HP and Microsoft jointly support the data warehouse appliance across all support service levels. Customers can achieve greater business benefit from the proactive services delivered by HP Critical Service and Microsoft Premier Mission Critical service, which are designed to minimize the negative business impact of downtime and to provide the highest-level support experience for mission-critical environments. In addition to joint support services, HP has worked with Microsoft to provide data warehousing consulting services through a very large combined worldwide staff of highly skilled professionals. These professional services include services for project steps such as assessment and business discovery, architectural design and data migration assessment, proof-of-concept development, and deployment of a data warehouse in a massively parallel architecture.

#225855

2010 IDC

OPPORTUNITIES AND CHALLE NGES FOR HP AND MICROSOFT

Opportunities
Joint HP and Microsoft BI and data warehousing solutions are likely to get a positive reception from large organizations that see value in this technology but face growing budgetary constraints. The various prepackaged options for the BI and data warehousing technology will provide large organizations with solutions that are easier to acquire, faster to deploy, and less expensive to support.

Challenges
At the same time, HP and Microsoft will face certain execution and competitive challenges, which are not unique to any particular vendor. On the technology side, one of the keys to the vendors' success will be a highly coordinated post-sales support process that needs to provide customers with a single point of contact to address any ongoing support issues.

ESSENTIAL GUIDANCE
There are three key issues that all organizations need to consider when evaluating data warehousing solutions: Does the solution solve a business need? Does the solution have the appropriate functionality and performance to address current and foreseeable future needs? What is the total cost of ownership of the solution including the initial purchase price as well as the ongoing maintenance costs for technology, staffing, and external services?

Solving the Business Need

The key to ensuring the success of any data warehousing project is for IT to make certain that the technology is being deployed in response to specific business needs. A decade ago, too many data warehouse projects were considered failed investments not because of IT's lack of technical expertise but because the projects, while elegantly designed, failed to take into account end users' evolving needs and time frames for information and decision support requirements. By now there is no question that the "if we build it, they will come" strategy does not work. When IT groups deploy data warehouses without direct business end-user input, they find that these technology deployments remain idle or substantially underutilized. Much has been written about the need for IT and business to communicate better, but not enough has been said about how to accomplish this nontrivial goal. The writer George Bernard Shaw said, "The single biggest problem in communication is the illusion that it has taken place." Too often, both parties make assumptions without fully engaging with the other side in a meaningful discussion.

2010 IDC

#225855

IDC's research into best practices suggests that IT groups tasked with data warehousing projects need to take on the role of a partner in their ongoing interactions with business end users. At a minimum, they need to become active listeners rather than passive order takers. This point was highlighted in a recent IDC discussion with an IT executive of a large organization known for an effective relationship between business and IT. In response to the commonly accepted notion that business end users are customers of IT (the implication is that IT serves business users), this CIO noted that business and IT are partners serving the same external customers together. Poor decisions by business users impact business as much as IT and reflect poorly on the whole organization. Therefore, the relationship must be based on partnership. Most IT groups are too focused on end-user requests for information without questioning the end users. Our recommendation is that rather than asking end users what their data requirements are or, worse yet, what their data warehousing requirements are, IT groups should be asking end users questions such as these: What are our organization's goals, and how do we measure progress toward these goals? What key performance indicators (KPIs) exist to ensure that measurement of progress toward organizational goals is made possible? What are the types of decisions being made at different levels of our organization? We find that leading organizations evaluate end-user decision-making processes, not simply data requirements. In other words, they don't ask "What data do you need?"; rather, they ask, "What decisions do you make?" They do so through a combination of interviews and systems monitoring, thus capturing information about ongoing decisions to help uncover opportunities for automating certain decision-making steps. Having said that, we note that the iterative nature of decision making and data warehousing projects makes the end-user requirements gathering process difficult. A common characteristic among leading organizations is the extensive use of rapid prototyping, which can be enhanced with an integrated technology solution that enables IT to focus on the customer rather than the technology integration. Although it is impossible to accurately predict all the future requirements, it is important to establish a strategy and a starting point for any data warehousing initiative and ensure that projects proceed in an iterative fashion as end-user needs and technology requirements evolve. An appropriate use of data warehouse appliances and reference architectures can provide organizations with the necessary tools to support an iterative data warehouse solution deployment plan.

Functionality and Performance

IDC research indicates (as shown in Table 1) that the top benefit of a data warehouse appliance that provides a prepackaged and optimized software and hardware solution is better query performance over a solution that is developed in-house from separate components.

#225855

2010 IDC

T ABLE 1
Benefits of a Data Warehouse Appliance Solution
Q. What benefits have you experienced or do you expect to experience by deploying a data warehouse appliance? (Select all that apply.) Benefit Better performance in query execution/scalability Faster time to deployment Better customer service from a single IT vendor Better availability characteristics Lower ongoing maintenance costs Lower initial price than separately buying software, storage, and server n = 308
Source: IDC, 2010

% of Respondents 27.6 26.0 21.9 21.4 18.4 16.3

Other requirements that fulfill the technical needs for scalability, availability, and security are just as important in the context of an optimized data warehousing solution. In evaluating specific current and future data warehousing needs, all large organizations should address the following questions when discussing their needs with technology vendors: What is the expected amount of data to be extracted from operational systems and other sources? The amount of data and its expected growth rate will determine the data warehouse capacity requirements and which specific integrated solution is most appropriate as the starter package. What is the expected frequency of loading data into the data warehouse? The current and expected frequency of data loads (along with volumes of data to be loaded) will be another determinant of the performance requirements of the data warehousing technology. What are the expected types of queries that managers, analysts, and other employees are likely to run? They may be simple queries that access a single database table and return basic descriptive statistics about a KPI, such as what were the sales last month or which of my customers bought the most products during a period? They may also be complex queries that require linking multiple tables and involve advanced predictive analytics to help evaluate the correlation between the purchasing patterns of two products or the likelihood of a customer buying a product or abandoning a service.

2010 IDC

#225855

What is the number of users who require access to the information from the data warehouse? Peter Drucker, one of the most well-known management consultants, said, "Most discussions of decision making assume that only senior executives make decisions or that only senior executives' decisions matter. This is a dangerous mistake." Not all employees are analysts whose primary role is to slice and dice data. However, every employee makes decisions, whether strategic or tactical, and therefore, organizations should ensure that appropriate access to the relevant information from the data warehouse is granted to staff at all levels. What would happen if the data warehouse was out of service? IDC research indicates that 25% of organizations would experience material negative impact on operations if the data warehouse was out of service for more than an hour, and over 60% said the same if the data warehouse was unavailable for up to one day. A growing number of organizations are highly reliant on the information from the data warehouse, which dictates the availability requirements of such a system.

Total Cost of Ownership

IDC research results shown in Table 1 also indicate that the second highest benefit of a data warehouse appliance is faster time to deployment over a solution that is developed in-house from separate components. Additional top-ranked benefits are better customer service from a single IT vendor (usually one of the vendors takes the lead in addressing the first level of customer contact), lower ongoing maintenance costs, and lower initial price. In addition to the technology evaluation, all large organizations should address the following questions regarding the total cost of ownership of the data warehousing solution: What types and levels of support are provided by the IT vendor or vendors? Does one of the vendors take the lead in the support process to ensure that the communication process is streamlined? What is the expected direct technology maintenance cost of the data warehousing solution? What types of internal data warehousing IT skills exist? This will influence the decision on the type of data warehousing technology to purchase as well as the appropriate balance between internal and external/consulting IT resources. By considering the above guidance, large organizations will be well served in their data warehousing solution selection processes, which should take into account the currently available integrated software and hardware solutions.

Copyright Notice
External Publication of IDC Information and Data Any IDC information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. Copyright 2010 IDC. Reproduction without written permission is completely forbidden.

10

#225855

2010 IDC