This action might not be possible to undo. Are you sure you want to continue?
Prepared for Sybase, Inc. by Judith R. Davis
Copyright © 2006 B-EYE-Network.com
. . . . . . . . . . . . . . . .com All Rights Reserved 2 . . . . . . . . . . . . . . . . . . . . . . . . . .6 Evolution of Business Intelligence Applications . . . . . . . . . . . . . . . . . . . . .4 What is Right-Time Business Intelligence? . . . . . . . . . . . . . . . . . . . . . .15 Sybase IQ . . . . . . . . . . . . . . . . . . . . . .11 Implementation Advice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 The Customer Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Summary of Features Most Important to Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Summary of Benefits/Return on Investment . .9 Organization Background . . . . . .14 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 The Right-Time Business Intelligence Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Introduction . . . . . . . . . . . . . . . .4 The Business Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Case Study: Shopzilla . . . . . . . . . . . . . . . . . . . .16 Copyright © 2006 B-EYE-network. . . . . . . . . . . . . .9 The Business Problem . . . . . . . . . . .14 About Sybase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Right-Time Business Intelligence TABLE OF CONTENTS Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 The Three Components of Latency . . . . . . . . . . . . . . . . . . . . . . . . . .15 Other BI solutions . . . .14 Conclusions .5 Right-Time Business Intelligence Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Company Background . . . . . . . . .
the company recommended answering a series of questions to address the implications of the decision on the BI environment. and making the Sybase IQ data warehouse more accessible to less technical users. The major benefit to Shopzilla and its merchants is the ability to react quickly to frequently-changing business environments. Right-time business intelligence encompasses any BI data collection. In this report. Emerging trends are centralized BI environments and a single view of the business. fast response time on queries while concurrently updating the database in near real-time. Right-time falls between intra-day and real-time Sybase IQ meets Shopzilla business requirements Key steps in implementing right-time BI Impressive benefits and ROI A solid foundation for the future Trends centralize BI information and encapsulate BI functions 3 . Once the decision is made. or action that falls between once-a-day and real-time. Other benefits include enabling merchants to customize their consumer surveys with minimal technical support. This is particularly critical in the online world. Shopzilla is a well-known online shopping portal.Right-Time Business Intelligence EXECUTIVE SUMMARY The right information delivered to the right people at the right time Right-time business intelligence (BI) is all about delivering the right information in the right format to the right people at the right time for decision-making purposes. Right-time BI applications have evolved over time to focus on operational business intelligence. analysis latency. Right-time business intelligence is used to run an organization’s daily business with the ability to modify the business intra-day. Shopzilla emphasized the importance of the decision to implement near real-time updates. and action latency. and encapsulating BI functions as common. reusable services. Shopzilla’s right-time BI solution provides a solid platform for future enhancements to extend the value beyond the current implementation. The study results highlight the significance of right-time business intelligence as another operational system that adds value to the business. There are three segments of business intelligence latency: data latency. we present an in-depth case study describing the right-time BI solution implemented by Sybase customer Shopzilla to improve the effectiveness of business decisions. exploring new BI platforms. Right-time BI optimizes (“right-sizes”) the time latency between when a business event occurs and when an appropriate action is taken. and simplification of the overall Shopzilla BI infrastructure. this usually means shortening the overall decision cycle. One area we explored in the case study was key steps in implementing a right-time BI solution. Examples include tripling the size of the data warehouse and adding many new users. Shopzilla also identified a significant return on investment (ROI). Benefits of Shopzilla’s right-time BI solution were impressive. Technologies for this include near realtime updates of data-warehouse data from operational systems and the packaging of BI capabilities as services available on demand. automation of survey processing. availability of more historical survey data for analysis. the integration of BI functionality within day-to-day business operations. The company uses Sybase IQ to capture and access merchant consumer survey data in a data warehouse. In business intelligence. analysis.
what do I already know about that customer and how can I use the information to serve the customer better? And look really smart because I have the customer’s profile at my fingertips. or compressing it. It doesn’t necessarily mean minimizing the time lag. If you need these pieces of information to solve a problem—say. “Well. global marketplace. and this is your second call about this problem…” Detailed data about the customer is transformed and integrated into useful information on which to make effective. Customers have high expectations. The point here is the importance of knowing what information you need and when you need it—what is the right information and when is the right time to look at it—and then ensuring your business can deliver it. optimizing the decision cycle typically means shortening it. If you need a piece of information now and you can’t get it now. and even news stories in addition to the traditional illumination of financial reports. and taking appropriate action based on that knowledge. We’ve been talking about this for decades—getting the right information to the right people at the right time in the right format. in addition to customers—we have the foundation for “righttime” business intelligence (BI). or automatically assuming that every decision process must be completed in “real-time. highly-publicized customer-satisfaction surveys. Companies are under more spotlights than ever before—new government compliance regulations. it may not be necessary or worth the cost. you don’t have it. The goal is to “right-size” the decision-making cycle. a prerequisite for survival and success in today’s competitive. Even if an organization can afford real-time decision-making. It is extraordinarily expensive to create a completely real-time organization. According to a recent article by Richard Hackathorn. If we extend this approach to all aspects of the business—suppliers. Right-time doesn’t mean real-time 4 . there is a shrinking margin for error and little tolerance for lack of good business intelligence. one that reflects business realities and the trade-offs between risk and cost. everyday decisions at every level of the business. you don’t have it when you need it. products. Mr. you are at a distinct disadvantage. WHAT IS RIGHT-TIME BUSINESS INTELLIGENCE? …to make smart business decisions… …at every level of the organization Optimizing the tradeoff between time and business value Right-time business intelligence optimizes the time latency between when a business event occurs and when an appropriate action is taken. Companies are serious about achieving this goal in order to be smarter about the way they do business. Smith. right-time takes into consideration the potential trade-off between time-to-action and the business value of the action. If you need a piece of information and you don’t have it. And if your competitor does have this information about its business. however.” The key is to define the “right” time for each decision cycle. Who are my customers? Which ones generate the most value for the company? What do my customers want from a relationship with my company? When a customer calls me.1 Making a decision today may have greater or lesser value than making the decision next week. I see that the red widget you are calling about was purchased within the past month from our downtown store. the time lag between knowing what is happening based on internal and/or external events.Right-Time Business Intelligence INTRODUCTION THE BUSINESS OBJECTIVE The right information at the right time… When it comes to intelligence about your business. In the BI arena. the problem statement is simple. plus the context I need to decide how to handle this particular call from this customer. etc. figure out why your product isn’t selling in certain markets—you aren’t going to be able to solve the problem.
Our goal was to understand the real-world challenges of right-time business intelligence. Oh. Current trends include: • Moving data from operational systems into the data warehouse more often than once a day or on a continuous basis. What are our options? We define right-time business intelligence as any BI data collection.g. The case study below describes the specific business problem.Right-Time Business Intelligence State-of-the-art used to be daily updates The state-of-the-art in business intelligence for a long time was a data warehouse (DW) and/or data marts updated overnight (within the traditional “batch window”) with data from legacy operational systems. The strategic direction here is to seamlessly integrate BI functionality into business processes. the right-time BI solution. multidimensional OLAP cubes).” Over the past five years or so. we also interviewed Shopzilla. 5 . It is BI used to run an organization’s daily business with the ability to modify the business intra-day. of the business. This recognizes the importance of business intelligence as yet another “operational” system that adds value to the business. benefits achieved. • Integration of data to provide a current “single view” of the customer. implementation advice. etc. with up-to-the-minute business context for analytical information. analysis. or action that falls between once-a-day and real-time. and future directions. • Extending access to analytical tools and information across the organization. The goal is to support intra-day analysis of up-to-date information with the ability to make immediate decisions about the business. how the solution was developed. return on investment (ROI). analysis. organizations have explored technology to support more real-time data collection. and by the way. enabling the retail buyer in the store. Organizations can shorten the decision-making cycle in many ways to facilitate right-time business intelligence. action Right-time trends BI is an operational system that adds value Understanding real-world customer experiences For this report. we describe the evolution of right-time business intelligence solutions. THE CUSTOMER PERSPECTIVE Businesses want more real-time BI Right-time is intra-day data collection. and loaded it into the data warehouse. or a dashboard to take advantage of righttime BI functionality. The overnight updates extracted operational data in batch. a Sybase customer that has implemented a righttime BI solution to improve the effectiveness of business decisions and processes. features critical to success. the call center operator. three are platinum frequent flyers with over five million miles each. and what lessons were learned. however. denormalized data. analysis. • Encapsulating BI functionality as reusable services that can be called “on demand” by any application.. and decision-making in a BI environment. Example: Flight 876 has been cancelled because of mechanical problems and we need to re-book all the passengers within the next 15 minutes. This approach still works for businesses where analysis of daily data is “right time. In the next section. transformed the data into a format for analysis (e.
The operational BI space is where right-time business intelligence fits. The analysis is usually geared to achieving a strategic BI objective. this brings the benefits of business intelligence to a broader population within an organization.Right-Time Business Intelligence RIGHT-TIME BUSINESS INTELLIGENCE SOLUTIONS EVOLUTION OF BUSINESS INTELLIGENCE APPLICATIONS Traditional BI focuses on strategic and tactical uses Let’s take a quick look at how business intelligence applications have evolved to support different types of decisions and analytical time frames. Another important direction is using a service-oriented architecture (SOA) to package BI capabilities as services available “on-demand” to business processes. enabling the organization to constantly modify its business within the day. BI analytics become a service available when needed. Operational business intelligence uses business intelligence in running an organization’s daily business. These BI services can then be linked directly into operational processes if appropriate. The analytical and decision-making time frame here is intra-day. or embedded within an operational process. and the analytical time frame involved is typically weeks or a few months. For example. Business intelligence is no longer a tool only for power users or analysts. The analytical time frame is typically 6 months to 1 year. Examples of near real-time business intelligence are credit-card fraud detection and financial risk analysis applications. Traditional approaches to business intelligence include: • Strategic business intelligence—Strategic use of business intelligence focuses on the achievement of long-term organizational goals such as increasing revenue or decreasing cost. An example is monitoring sales during the day. The ultimate in right-time business intelligence is near real-time business intelligence. where data is analyzed as soon as possible after it flows into the organization. analysis. People and processes in operational departments can now take advantage of the power of business intelligence. creating a closed-loop process. or ways to optimize a wider range of decisions by integrating BI capabilities within business operations. • Tactical business intelligence—Tactical use of business intelligence addresses interim. Traditional uses of business intelligence are supported by data warehouses and data marts that are updated daily (or less frequently) using overnight batch windows. decisions Organizations are now exploring operational business intelligence. the concept of the overnight batch window for updating the data warehouse may be replaced by the need to trickle-feed data changes into the BI system during the operational day. The technologies for implementing operational business intelligence are different from those used to implement traditional BI applications. Operational BI focuses on intra-day updates. 6 . Ideally. shorter-term organizational goals such as analyzing a marketing campaign for a specific product. Integrating BI within business operations The goal is to close the gap between analytical applications and operational applications.
extraction. relates ongoing data events or changes to business rules. delivery. prepare it for analysis. the more eventdriven it becomes. Inc. validation. and others. • Event-driven business intelligence—An event-driven BI application monitors specific business operations.2 7 . There are a wide variety of tools and products that address one or more of these aspects of data latency. analysis. EAI. transformation.1 Data latency is the time it takes to collect raw data. and loading. Bolder Technology. The BI application may also have the ability to take action itself based on knowledge embedded in the system. enterprise information integration (EII). event-driven.Right-Time Business Intelligence Another way to categorize BI applications is whether they are event-driven or demand-driven. right-time BI monitors the health and well-being of the organization. transform. cleansing. For more information on these technologies and which to use when. the system alerts someone and may include recommendations about what action should be taken. There are three segments of latency on the continuum between when an event occurs and when an informed action is taken—data latency. integration. and generates alerts when appropriate. Important functionality here includes data profiling. including extract. When exceptions occur.) This concept was developed by Richard Hackathorn. The destination platform of the data is typically a data warehouse or data mart. and store it where it can be accessed and analyzed. and decision points Another context for right-time BI solutions is the component of the decision-making process each addresses. data replication. and EII Tools to Create an Integrated Enterprise” by Colin White of BI Research. These tools fall into several categories. and load (ETL). please see the Data Warehousing Institute (TDWI) research report titled “Data Integration: Using ETL. analysis latency. • Demand-driven business intelligence—A demand-driven BI application is accessible to operational systems when analytical data is needed to make a business decision. enterprise application integration (EAI). or the system may make a decision automatically. and decision/action latency (see Figures 1 and 2 below. Thus. An example of demand-driven business intelligence is a call center operator firing off a query against a federated database to get information about a customer on the phone. master data management (MDM). THE THREE COMPONENTS OF LATENCY Latency occurs between data collection. The closer an organization gets to real-time. transformation.
and generate alerts if appropriate. The Benefit of Reducing Latency Figure 2.Right-Time Business Intelligence Three Components of Latency Figure 1. Reducing latency at one or more points in the decision-making time continuum can dramatically increase the business value of the decision Analysis latency is the time it takes to access the data. apply business/exception rules. Analysis may be done by a user or an application. The elapsed time may result in lost business value to the organization. analyze the data. 8 . turn the data into information. This shows the three components of latency between when a business event occurs and when action is taken.
After raising enough venture capital. Only then can the organization assess which BI technologies and products can be applied as part of a right-time solution. Its Web site is designed to help Internet shoppers quickly find the products they want at the best price. In the case of new compliance regulations.. Shopzilla also provides shopping content for other Web sites. Survey types include: • An online point-of-sale (POS) survey—This survey collects customer feedback on the first part of the online ordering process: placing the order. companies often need to report exceptions as soon as possible. the company expanded into comparison shopping in the early 2000s. based on knowledge of the business. and Anish Balakrishnan. In addition to the U. features influencing the purchase. Right-time is businessand process-specific Implementing right-time business intelligence involves detailed analysis of each business process to look for opportunities to shorten the decision cycle. Shopzilla continually updates its database of merchants. CASE STUDY: SHOPZILLA ORGANIZATION BACKGROUND A leading online comparison. Specific metrics addressed include evaluation of the online ordering process. France..000 merchants in 2. up-to-date search results using its proprietary ShopRank search engine.Right-Time Business Intelligence Decision latency is the time it takes to receive an alert.shopping service Shopzilla is a well-known. Surveys serve two purposes: (1) enable Shopzilla to rate merchants and (2) enable each merchant to improve its business by directly analyzing its own survey results. Shopzilla is now an operating subsidiary of E. review the analysis. The number of employees is 210 and growing. such as AOL. Shopzilla is headquartered in Los Angeles. the company initially focused on collecting consumers’ ratings of their online shopping experiences. Shopzilla conducts customized surveys for many of its merchant partners—Circuit City. The company name was changed to Shopzilla in late 2004.com. and demographic information.500 categories. Staples. in addition to its shopping search engine. Lycos. and publishes the results of these surveys. The BizRate market research division summarizes. Director of Database Administration. and Time Warner’s RoadRunner. analyzes. and prices to provide each site visitor with relevant. 9 . and take action. and Froogle. Director of Strategic Projects. THE BUSINESS PROBLEM Global reach and influence Consumer surveys provide feedback for merchants and Shopzilla A major component of Shopzilla’s business. products. Handling huge volumes of frequently-changing product data Shopzilla’s Web site currently offers access to over 30 million products from more than 60. and others. for this case study. For example.W. a diverse media conglomerate with businesses in areas such as newspaper publishing and broadcast TV. Founded in 1996 as BizRate. NexTag. Shopzilla also conducts surveys on its own behalf. Shopzilla’s 2004 revenues were $67. if any. is customer surveys. Yahoo! Shopping.com. online comparison-shopping portal. Competitors include Shopping. and Germany. Scripps Co. among others.S. and these volumes are steadily growing. overall satisfaction and loyalty. decide what action is required. manufacturing companies with just-in-time inventories need updates throughout the day. We interviewed Ranga Srinivasan.4 million. Shopzilla has country-specific versions of its Web site for the United Kingdom.
Right-Time Business Intelligence • An online fulfillment survey—This survey collects feedback on the second part of the online ordering process: delivery and post-purchase support. Metrics here are similar to those of the POS survey with a focus on post-purchase service and support. Shopzilla also wanted to simplify the overall infrastructure and provide compatibility with its Web operation. provides merchants with valuable customer feedback. including schema changes. Shopzilla wanted to offer merchants (1) the ability to completely customize their surveys based on unique business requirements. and (3) near real-time access to their data with no technical investment on their end. purchasing behavior. and the ability to automatically add columns and tables on-the-fly to capture custom survey questions. Internet usage. including trends in shopping behavior and awareness/effectiveness of marketing and advertising. Another key goal was to automate survey processing to minimize resources and time required to update the database of survey results. The company planned to store at least 18 to 24 months of data in the data warehouse. Shopzilla updates the merchant-rating information displayed on its Web site. and more flexibility in addition to reliability and high availability. To do this. • An online panel survey—Shopzilla maintains a panel of over 800. Shopzilla realized that its solution for survey data was too static and would not support future requirements. In this case. pricing and a low total cost of ownership (TCO) was a big issue for the company. These surveys can be targeted to distinct audiences based on demographics. Shopzilla developed a daunting list of strategic business intelligence requirements to achieve these goals: Technical goals: 100% uptime. • Support for a large number of columns per table—Shopzilla anticipated the need for upwards of 1. And last but not least. and merchant surveys were not customizable. • An offline survey—Customers who buy something at a physical store are invited to provide feedback on the shopping experience via a secure URL. The panelists are called on to answer a broad range of questions. query response was slow. Given its competitive environment. Data volume was growing fast and there was a need for ad-hoc reporting. 10 . Offline survey results are treated separately from online results.500 columns in its main survey tables. ad-hoc reporting was not supported. (2) the opportunity to improve their business based on ad-hoc analysis of 18 to 24 months of survey results. the buyer receives an email from Shopzilla with a link to the survey. fast update and retrieval Flexibility to customize survey content • 100% uptime—The business intelligence system needed to be available 24/7. • Instantaneous query response time for ad-hoc reporting—Shopzilla wanted fast query response time for both merchant-survey analysis and internal analysis. From survey results. Results of fulfillment surveys are tied to POS surveys for the same consumer. • Ability for merchants to customize surveys without requiring technical assistance—Shopzilla wanted to give merchants the flexibility to design their own survey questions. The customer logs on to a Web site to complete the survey. data was at least one-day old. Survey data was stored in a Sybase RDBMS. • Near real-time data updates—Shopzilla wanted to update survey data in the data warehouse at least every 15 minutes while supporting concurrent ad-hoc queries.000 buyers who can provide merchants with immediate answers to critical e-commerce questions. and publishes its own market-research reports. A major business goal was to enhance the value of the Shopzilla relationship for the primary customer—the merchant—and develop a closer connection with merchants. The challenge was to find a solution with both hot backup and no downtime for any maintenance activities. etc. 365 days a year. Current data warehouse was too static In 2001. faster response times. regardless of the complexity of the query and the amount of underlying data.
reformat the flat files. highly-optimized indexes on each column to capture averages. When Shopzilla updates the database. the script would clear the answer as incorrect. “We would like to do this more frequently but can’t because query performance degrades to an unacceptable level. Shopzilla already has the customer’s order number. Surveys are divided into pages. a survey can have several pages. email address. and other identifying information (even if the customer cancels while in the middle of taking the survey).600 columns.5 on Sun servers using the Sybase IQ Virtual Backup strategy. Shopzilla also builds multiple. • Handle increasing volumes of data—Shopzilla also knew that as the number of merchant clients increased. delete. such as “Where did you hear about us?” The POS online survey table is the largest fact table in Sybase IQ. The survey questions come from the ASE transactional database to ensure that all customers get the same survey from each merchant. This is where Shopzilla needs near real-time updates and fast response time on ad-hoc queries and reports. THE RIGHT-TIME BUSINESS INTELLIGENCE SOLUTION Sybase IQ handles survey results and ad-hoc reporting Shopzilla’s right-time business intelligence solution is focused primarily on processing and analyzing survey results. An example is a question that can have more than one answer. For certain types of new questions. This is used for internal reporting on more static financial and historical data. the company implemented Sybase IQ version 12. etc. Based on the type of question. The data warehouse had to adapt seamlessly to such growth without adversely impacting query performance and near real-time data-refresh rates. if someone says he is 13 years old and has four children. Microsoft SQL Server. with as many as 1. and Oracle. it is saved on the server in a flat file. As the customer completes each page. To handle the collection of survey results and the merchant reporting system. Shopzilla then enters the customized survey data into the Sybase ASE system and the survey goes live when scheduled. The solution also had to support Transact-SQL and provide easy data migration from Sybase. and load the data into the Sybase IQ survey database. ASE feeds data overnight to Sybase IQ via flat ASCII files. the change is immediately reflected in the survey. For example. Survey results are updated every 15 minutes Columns are added onthe-fly for new survey questions Shopzilla also designed a rule-based system to automate the process of adding new survey questions to the database schema. he/she is connected to a Shopzilla server.Right-Time Business Intelligence Adapt to growth • Ability to easily migrate to a new database structure—Shopzilla realized that a columnoriented database was the best architecture. Merchants customize their own surveys NEW SURVEYS. the script decides what type of column is appropriate for storing the data and creates the column on the fly if necessary as the data is loaded into Sybase IQ. or modify questions. As a note. Shopzilla provides a standard survey form that the merchant customizes.” says Balakrishnan. 11 . A pre-script checks survey answers for data consistency and validation. with five questions on each page. For a new survey. Shopzilla’s environment also includes Sybase Adaptive Server Enterprise (ASE) for transactional data. Shopzilla also maintains a traditional data warehouse in Oracle that is updated once a day from flat files and ASE database tables. especially during the holiday season when daily data volumes increase significantly. COLLECTING SURVEY RESULTS. Another post-script runs every 15 minutes to collect all completed survey pages (even if the customer hasn’t completed the entire survey). there would be a resulting increase in data volumes and database size. Merchants can request Shopzilla to change a survey at any time to add. When a customer takes a survey. min-max data. the script creates a table.
schema changes to add columns to tables. As a note. overall customer satisfaction. The merchant might also decide to take action to try to retain the customer with a special offer. This gives the merchant useful information about ways to improve the business. The merchant can make projections for a category. queries were terminated. So Shopzilla installed the Simplex version of Sybase IQ.Right-Time Business Intelligence MERCHANT REPORTING SYSTEM. The business intelligence environment also includes ad-hoc queries and reporting on the inventory of 30 million products. Performance optimization was key 12 . to maintain data consistency. Access to customer comments provides context The merchant can see exact customer comments. To handle this. Second. may be able to match a comment to a specific order. An additional merchant benefit accrues from Shopzilla’s partnership with Coremetrics. look at trends. and multiple readers. Shopzilla’s “quick response” feature sends comments from Sybase IQ out to the merchant as soon as they are received. For example. Standard filters are retail versus business customers and date ranges. The company faced two significant technical issues. “This is a reporting tool for the merchant. and compare results with category and industry aggregates.6). adding perhaps two seconds to each query in the worst case. Shopzilla provides a secure. updates. Shopzilla also uses IQ for internal analysis INTERNAL ANALYSIS. product type. with frequent data loads. The goal was to handle a mixed workload of data updates. etc. and in some cases. (It does in version 12. It is very important to assign every product to the most relevant category to ensure customers find what they want. resulting in unacceptable performance. the Multiplex version of Sybase IQ did not support schema changes on the fly. and can apply filters to the data.g. sex. Suppose a customer says “No” to the question “Would you shop here again?” and gives details about why. This eventually leads to a situation where Sybase IQ runs out of memory and begins paging to disk. The merchant can apply up to four additional filters to slice and dice the data for each query. a script runs every five minutes to kill idle connections to the database. In the 2001-2002 implementation timeframe. age ranges. these “old” versions proliferate. something that can even be taken into a board meeting to show quarterly or annual results. marital status. Schema changes don’t take much time. Available filters are based on the questions on that survey (e. and reader processes. Shopzilla ties Coremetrics data to Shopzilla survey results to add context for merchant analysis. taking up memory space that could be otherwise used for processing ad-hoc queries. THE PERFORMANCE CHALLENGE. Coremetrics software analyzes Web site traffic and captures and stores all customer and visitor clickstream activity to examine the causes of site abandonment. Sybase IQ creates a version of the data—a snapshot—for each reader based on the last checkpoint.” states Balakrishnan.). A major challenge for Shopzilla was figuring out how to load data into Sybase IQ without impacting anyone reading the data. However. Results are displayed as graphs. online environment where a merchant can run ad-hoc reports to analyze its survey data. Shopzilla explicitly issues “rollback” commands after every SQL query to release read locks on each table accessed. Shopzilla uses Sybase IQ to analyze how products are categorized. In addition. etc.. This is a fairly recent application of business intelligence. A system of “handshakes” (using semaphore locks) between a write process and reader processes ensure that reader processes on a table wait until the write is complete. view volume growth. Shopzilla developed its own technology to efficiently interleave schema changes. identify days with the highest sales volume. One was the fact that when an update transaction on a table locked out readers. The merchant sees a list of questions on the survey.
Having current information about customer experiences is critical to identifying the potential need for corrections and the ability to take timely action to maintain or improve satisfaction.000 characters each] which are a huge problem to handle in our data warehouse environment. Investment supports dramatic growth 13 . highlyoptimized indexes. consider the following: • What is the projected total data volume and what percent needs to be near real-time? A high volume of data with a small percentage in near real-time is very different from a situation where most of the data needs to be near real-time. To achieve fast queries with frequent updates.. “We get huge comments [up to 8. takes more than 10-15 seconds. We store text comments in a file-based system and have a highly-optimized way of retrieving them. Every query is restricted to a maximum of one year of data. it has maintained performance even though the volume of data has tripled and the number of users has increased dramatically. not even the most complex ad-hoc query. The results (graphs) of both queries can be viewed together. consider solutions that offer data compression. keeping the TCO low.Right-Time Business Intelligence To achieve its goals. this is a nice-to-have. The biggest issue is retrieving the text from the database. Merchants now have access to 18 to 36 months of survey data instead of just 13 months. the company purges unneeded historical data as often as possible. • Does the system need to support ad hoc queries? If yes. The effort and costs involved are significant enough to really question the necessity of this. For example. The first and most important question is: Do you need near real-time data? “Too often. And Shopzilla has not had to significantly increase its investment in hardware and software. ROI. A year’s worth can be a lot of text.” Once an organization decides to go with a near real-time solution. look for a solution with highlyoptimized indexes to make queries as efficient as possible. and controls user access to data through an application. customer satisfaction can make you or break you. an organization may have to place limits on how users access the data. looking at two years requires two separate queries. • How much data is text-oriented? Shopzilla surveys can allow the consumer to enter comments.” SUMMARY OF BENEFITS/RETURN ON INVESTMENT Performance tradeoffs Ability to identify changes and react quickly BENEFIT. The major benefit of right-time business intelligence to Shopzilla and its merchants— ad-hoc query and reporting on near real-time survey data—is the ability to react quickly to frequently changing business environments.g. Shopzilla has combined these and other optimizations with regular data purges and Sybase IQ’s native performance functionality (e. data compression. Shopzilla wanted instantaneous query response with updates every 15 minutes on a database containing two years of data. This is a trade-off Shopzilla made to get both query and update performance. • What is the projected growth of the data? If it is high. Since Shopzilla went live with its right-time business intelligence solution in mid-2002. No query. • What is the average query response time desired? There is always a performance trade-off in a mixed environment of queries and near real-time updates. In the online world. detailed query plans) to ensure users get the desired fast response time. IMPLEMENTATION ADVICE Understand the issues with near-real-time updates Balakrishnan and Srinivasan recommend answering a series of questions before considering a business intelligence solution based on near real-time data updates. To achieve this.
three-year effort has resulted in a high-performance. Sybase IQ offers these important features: • • • • • • • • • • • Instant query response time out-of-the-box The ability to quickly load data in batch The ability to have up to 45. produce an integrated. and in the format necessary. and the company is looking to move to the next level of capability. The raw data on which to make effective decisions exists in most companies. consolidated view of the business (including data. The solution is now very mature. An intense. FUTURE DIRECTIONS New platforms and technologies to handle future growth Shopzilla has spent the past two and a half years optimizing the Sybase IQ data warehouse. • Evaluating use of Sybase Replication Server to move data in near real-time from ASE to Sybase IQ. and processes for analysis and decisionmaking.000 columns in a single table Automatic data compression Low TCO Support for Sybase’s T-SQL provides compatibility with the ASE server The ability to build specialized indexes The ability to import data from Oracle and Microsoft SQL Server Easier to manage and maintain than almost any other DBMS The data warehouse can grow as the business grows (add more query servers to expand) Out-of-the-box high availability. both merchant partners and internal users. Shopzilla’s right-time business intelligence solution has enabled the company to satisfy a difficult set of business requirements and provide an important service to its growing community of merchant clients—fast and easy access to near real-time survey results. and make it available (either directly or as a callable service) to business users. This will enable Shopzilla to extend the scope of the Sybase IQ data warehouse with new applications. and organizational hierarchy). • Moving to the Multiplex version of Sybase IQ on Linux to enhance performance. effective business decisions. handle peak loads. 14 Right-time BI is a critical foundation for future growth .Right-Time Business Intelligence SUMMARY OF FEATURES MOST IMPORTANT TO SUCCESS Sybase IQ meets business requirements Balakrishnan states that Sybase IQ “gave us the groundwork on which we could build a solution that solved our problems. CONCLUSIONS The goal: all data is integrated and available Right-time business-intelligence highlights the state-of-the-art in developing BI platforms to support timely. Right-time business intelligence is and will continue to be a key factor for Shopzilla in achieving success in a highly-competitive. near real-time business intelligence platform that offers Shopzilla a solid foundation on which to grow the business. Future directions include: • Tripling the size of the data warehouse and adding many new users.” Specifically. and adapt to growth while keeping TCO and the cost of adding incremental capacity low. where. • Making the Sybase IQ data warehouse more accessible to less technical users through more intuitive interfaces such as the Excel-like SPSS client. The goal is to synthesize all available data (internal and external). applications. fast-moving business environment. business rules. business definitions. Shopzilla recognizes this and has implemented a solution to develop business intelligence and connect it to business-decision processes. It just isn’t available when.
embedded.Right-Time Business Intelligence The bottom line: It’s your business. and Sybase iAnywhere for mobile. SYBASE IQ Sybase IQ for business intelligence A real-time analytics server for highperformance query and reporting Sybase IQ is Sybase’s strategic analytics server for BI and DW environments. and small-to-medium business (SMB) support. and Sun Solaris. “The BI Watch: Real-Time to Real-Value. Microsoft Windows. Sybase is headquartered in Dublin. Sybase IQ’s performance was fast and queries did not impact the performance of the underlying transactional database.” research report published by The Data Warehousing Institute.6. The goal is to enable customers to optimize and enhance current investments. Sybase IQ was also designed to cache the indexes in memory where possible. of the data that would be needed to satisfy queries. Colin. cross-platform solutions for information management and delivery. Sybase IQ for business intelligence (BI) and data warehousing (DW). and EII Tools to Create an Integrated Enterprise. and healthcare. Sybase was the first RDBMS to incorporate intelligence in the DBMS engine itself with features such as triggers and stored procedures. The company provides open. In 1987. communications. and extend the reach of business-critical information to users on the front lines. Sybase IQ is designed to support complex analytics on large data volumes. Key industries supported are financial. and Oracle in addition to Sybase. Thus.” DM Review. and did not require disk I/O to look up data. Sybase IQ runs on HP-UX. Microsoft SQL Server. References 1 Hackathorn.com) is a global software vendor focused on managing information throughout the enterprise. including IBM DB2. Sybase IQ provides a real-time analytics server designed for very-fast query response on large volumes of data. Sybase has pioneered other RDBMS developments as well. 2 ABOUT SYBASE COMPANY BACKGROUND Enterprise-wide information management Sybase (www. queries ran primarily against the cached Sybase IQ indexes. Sybase Replication Server moves data among heterogeneous sources. Annual revenues were $789 million in 2004.sybase. link existing data resources. EAI. Your goal should be to get it right with right-time business intelligence. government. The idea was to build bit-mapped indexes on the base data with the option to include in the indexes most. Sybase IQ has a number of features that distinguish it from a traditional RDBMS. “Data Integration: Using ETL. Sybase IQ has since evolved into its own high-performance database-management server. 15 . October. January. White. As a result. The company’s family of database servers includes Sybase Adaptive Server Enterprise (ASE) for transactional environments. including an open architecture and data replication. Sybase IQ began as a bit-mapped indexing system to provide read-only access to data stored in the Sybase RDBMS. The latest version is 12. 2005. CA. Richard. 2004. if not all. IBM AIX. To achieve query performance. Informix. Linux.
in contrast. enterprise modeling tools. Here are some recent examples. OTHER BI SOLUTIONS To meet evolving customer requirements. • Large page size—Sybase IQ supports a page size of up to 512K to optimize I/O. This platform combines Sybase’s analytics server. The Risk Analytics Platform is aimed at capital markets applications. Sybase IQ would not have to do any disk I/O to return the results. where the data is integrated with historical data. For example. Dynamic Operational Data Store Dynamic Operational Data Store (ODS) is a horizontal solution for any customer that needs to execute queries on operational data without impacting operational performance. The ODS is dynamically loaded based on changes in the operational system. To evaluate multiple query criteria. Sybase IQ allows up to 45. such as Informatica. Keeping 10% of the data in memory would require 100GB of memory. is used to “dynamically” offload data from the operational system into a separate Sybase IQ “ODS” database for reporting and analytical queries. it is faster to read and write compressed data than to read and write uncompressed raw data. transform. not rows Bit-mapped indexes.000 columns per table. and integration with the JMS messaging bus from Tibco. Sybase IQ’s column-based architecture also adds flexibility to schema changes. to find all customers in a particular zip code. • Bit-mapped indexes—Sybase IQ’s bit-mapped indexing capability supports a variety of index types that can be tailored to the data type. The intent is to help traders analyze and spot opportunities faster and support near-real-time trading decisions. Sybase IQ creates a snapshot of a table whenever the table is updated. would compress 1TB down to 300 to 700GB on disk. The data model captures inbound “trades and quotes” from the major exchanges and updates the analytics server. Real-Time Data Services Risk Analytics Platform 16 . and a risk-analytics data model derived from industry best practices. keeping 10% in memory requires 30 to 70GB of memory.Right-Time Business Intelligence Data is stored in columns. such as adding or deleting columns. Sybase IQ. Sybase Replication Server or an ETL (extract. Sybase IQ overlays the bit maps for each column to find the intersection (and) or the union (or). It doesn’t have to read all of the data in every qualifying row to find the result. and cardinality in a particular column. • Multiplexing—Sybase IQ-Multiplex uses a shared-storage architecture for parallel processing across multiple servers. This makes retrieval extremely fast because Sybase IQ only has to access the data in the columns included in the query. For example. and load) product. Alternatively. For Sybase IQ. Sybase has developed solutions that combine multiple products and target particular industries or applications. • Data compression—Another feature of Sybase IQ is the fact that it compresses data by 30% to 70%. ASE triggers. compression allows Sybase IQ to store more data in the same amount disk space. data distribution. Sybase IQ also only allows one writer at a time per table. This solution takes pressure off operational systems while giving users query and reporting capability. Sybase IQ would simply read the zip code column (or index) to locate the qualifying rows. • Table-level versioning—To maintain query performance while data is loaded. large page size for fast access Database-level compression Versioning supports concurrent queries and updates • Column-based architecture—Sybase IQ stores data as columns on disk instead of as rows. And if all of the data requested in the query is included in the index. The goal is to reduce the latency between when a business event occurs and the ability to take action based on that event. a 1TB database could require as much as 3 to 5TB or more of disk space in a traditional RDBMS. Compression minimizes the problem of data explosion as database volumes grow by reducing the cost of both storage and memory for the customer. Real-Time Data Services combines Sybase replication technology.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.