WHITE PAPER

Delivering Real-Time BI with Complex Event Processing

Seth Grimes, Alta Plana

www.sybase.com

INTRODUCTION In todays business climate, time is money has never been truer. The pace of business is fast, operations are 24x7, physical borders have been erased, and the flow of information is constant. Economic pressures weigh on every business. Competitiveness hinges on knowledge and on timeliness, on delivering the right information to the right people and systems at the right time. A decision advantage relevant, complete, accurate, and timely data the ability to see and act on opportunities and vulnerabilities before the competition does is essential. Traditional data management and analysis solutions fail to deliver this advantage. Relational databases were designed for transaction processing, not speed-of-data decision making. Mainstream business intelligence similarly cant keep up with the variety of data sources sensors, market data, in-process transactions, Web clickstreams, and the like, many of them high volume that affect business operations and customer experience today. BI solutions that were designed for static, cleansed, historical data help you understand where your business has been. They dont provide for automated, real-time response to business opportunities and risks as they arise. The gap between traditional BI and real-time analytical needs is where complex event processing (CEP) and the notion of continuous intelligence support for a now, cross-source, 360-degree view of the operating environment come into play. CEP technology first emerged in response to highly demanding, highly focused trading, intelligence, and communications problems. Early, custom-coded applications were hard-wired for a narrow set of data sources and processing algorithms. These applications have been supplanted by commercial offerings that integrate filtering and analysis of high-volume data streams market-feed data rates, for example, edge into millions of ticks per second with the ability to consume messages from diverse sources, supporting generalized, predictive analyses. Commercial solutions include tools for both developers and end-user analysts. They utilize design studios, data schemas, query languages and utilities, and dashboard displays that will be immediately familiar to BI practitioners. The best of them allow businesses to embed front-line analytics in their operational systems. They build on BI foundations to deliver decision advantage to the broad business-analytics market. This white paper explores complex event processing and the extension of BI interfaces and applications to support event-driven, real-time, operational BI and automated decision making: continuous intelligence.

REAL-TIME, RIGHT-TIME BUSINESS DECISION MAKING Competitive pressures have created an analytics imperative, motivating adoption of business intelligence and data warehousing technologies that seek to extract business value from data assets. Traditional BI and data warehousing on their own, however, fail to cover the full set of modern-day operational needs. Their shortcomings have spurred organizations to move beyond BI to complex-event processing. Traditional BI is: Historically focused rather than operational and predictive. Designed only for monitoring, reporting, and interactive analysis rather than embedded, event-driven decision making. Highly specialized, for instance for performance management and financial analysis, and not pervasive. Similarly, the Load > Index > Query approach supported by traditional database management systems (DBMSes) and data warehousing cannot keep up with todays need for speed, for faster data and low-latency, real-time analytical processing. Processes and Interfaces: BI versus CEP CEP solutions build on traditional BI and data-warehousing techniques and interfaces, leveraging practitioners hard-won knowledge and experience. They enable a jump to real-time, right-time business decision making via familiar, proven BI and data-warehousing analytical processes, extended to data streams, service oriented architectures, and event-driven processing, as illustrated in Table 1.

ANALYTICAL PROCESS Data sourcing

BUSINESS INTELLIGENCE database tables; ODBC/JDBC/ Analysis Services sources ETL, ELT tools; in-database (data warehouse) integration SQL queries and functions; point-intime metric/KPI values record sets; operational applications must poll the database

COMPLEX EVENT PROCESSING databases, data streams, messaging systems, event clouds: via adapters and service subscriptions transformation and normalization of live data in flight continuous computations on live & historical data via extended SQL record sets and KPIs, output streams, business rules engine integration

Integration & cleansing Computation of metrics & indicators Output/results processing

Table 1: Complex Event Processing extends BI analytical processes.

Theres more to analytics than processes however. We are also concerned with the user view, hence the CEP-BI comparison offered in Table 2. USER INTERACTION Interfaces Enterprise integration BUSINESS INTELLIGENCE dashboards, reports, pivot tables: point-in-time values portals, developer kits, design studios COMPLEX EVENT PROCESSING event-enabled dashboards: backed by continuous computations portals, developer kits, design studios

Table 2: User facing/integration elements of BI and CEP.

Enterprise requirements do go beyond technical features and interfaces. To support the jump from business intelligence to continuous intelligence from static BI to continuous, event-driven analytics enterprises need the robustness, performance, scalability, security, and high availability delivered by enterprise-grade CEP systems. Understanding CEP Processes For maximum benefit, the two styles of analytics, BI and CEP, should be integrated into everyday business operations, BI for strategic decision making and CEP for tactical monitoring and for speed-of-data action. With BI as a foundation, we will look at CEP processes as they relate to operations. Data sourcing BI and CEP derive analytical value from data that affects business outcomes. BIs aim typically relates to improving organizational performance using transactional data extracted from corporate systems. CEP taps those transactional sources, but as the data is produced as well as via database and data warehouse queries. CEP accesses live data streams. Examples include time-stamped stock-market tick data with data tuples representing the trade/offer/bid price and share volume for a given security; telephone-call records, credit-card and bank transactions, and sales reported as they occur; Web-site interactions; and temperature, pressure, movement, and other physical conditions reported by sensors. Adapters provide the interface between stream and message sources and CEP engines. Integration & cleansing BI has long relied on extract-transform-load (ETL) for data integration and cleansing; extract-load-transform (ELT) with in-database transformations and disciplines such as master data management (MDM) have become popular. By contrast, CEP integration and cleansing are literally streamlined: the software filters and joins data from disparate database and stream sources as part of the continuous computation process. With traditional, BI-reliant data warehousing, data latency a lag between data arrival and readiness for analysis occurs when values are written to disk and indexed for queries. BIs use of batch updates and dependence on conventional data structures further creates analysis latency. CEP reduces both classes of latency to the millisecond level via in-memory processing and speed-of-data computations.

Metrics & indicators While BI algorithms can be pretty sophisticated, most computations involve aggregation (summing and counting) and simple calculations such as percentages. Key performance indicators (KPIs) are derived performance-management values that involve more complex calculations, and data mining adds a further sophistication by generating predictive models from data. CEP supports BI-style computations and can also accommodate predictive models. Both are extended with the ability to discern event patterns over sliding time and record windows to turn BIs static, point-intime metrics and KPIs into continuously computed, action indicators. Output/results processing The use of BI KPIs, reports, and analyses rarely goes beyond dashboard monitoring. CEP enhances BI-style monitoring and is also capable of driving automated, operational decision execution. In doing so, CEP responds to events that reflect actions or data-defined conditions: a Web-site visitor adds an item to her shopping cart, the inventory of product X falls below 50% of average weekly sales. The outcome is real-time business optimization, complementing BI and its strategic focus.

BI IN REAL TIME AND MORE Decision-systems expert D.J. Power describes business intelligence as a set of concepts and methods to improve business decision making by using fact-based support systems. Traditional BI tools, however, are hard-pressed to meet Powers definition in a world of always-on business. It takes CEP, building on the foundation of traditional BI, to keep pace with emerging operational analytical needs. Traditional BI suites rely on reporting, OLAP pivot table, charting, and dashboard interfaces to tease insights out of historical data. Some vendors offer operational and real-time BI solutions in an attempt to break out of a reactive straitjacket. Definitions and capabilities vary. Some define operational BI as simply reporting from operational databases. This class of operational BI is limited to monitoring current indicator values. It supports, for instance, basic business activity monitoring (BAM), but lacks capacity to integrate diverse sources and automate response to emerging opportunities. Similarly, real-time BI often simply means real-time scoring of static predictive models backed by very modest analytical capabilities rather than comprehensive ability to make sense of real-time operating conditions. Data-mining and predictive modeling extend BI capabilities, albeit incompletely. Data mining discovers patterns in data; the modeling exercise provides a tool for estimating future values of variables of interest or classifying new cases according to past experiences. Predictive analytics has applications for real-time credit scoring, financial-transaction fraud detection, compliance monitoring, and the like, but the analyses remain static, based on predefined models and conditions. These functions are forward-looking, but theyre still based on historical data. Predictive analytics and operational and real-time BI are definite advances over traditional BI, but it takes CEP to go beyond what D.J. Power calls pushing real-time information to decision makers, to supporting real-time, right-time business decision making, to progress from business intelligence to continuous intelligence. CEP works alongside and established techniques, adding the time and record windows, continuous computations, event-pattern matching, and operational embedding that move BI into real time to deliver continuous intelligence.

Real-time data: data stream and messaging sources

Complex Event Processing

uous Contin ce elligen Int tive Predic alytics An

Data Mining

Interactive Analysis (OLAP) Historic data: database and data warehouse sources Reporting Static query and analysis
predictive modeling > real-time, event-driven analytics

ss Busine ce n tellige In

Dynamic query and analysis

The path to continuous intelligence: static reporting > interactive analysis > predictive modeling >

THE PATH TO CONTINUOUS INTELLIGENCE The path to continuous intelligence starts with awareness of opportunity/risk gaps that stem from the inability of traditional BI to respond to complex, event-driven business conditions. A responsive solution architecture, bridging gaps, will cover both information technology and business processes. Proof-of-concept prototyping, an evaluation and implementation best practice, demonstrates solution practicality and is followed by phased introduction of new technologies and work practices. Gap Assessment Risk/opportunity gaps result from business scenarios that include: Failure to incorporate important business information, typically from untapped or underutilized data sources, in decision-making processes. Unacceptable delay reacting to significant business events that can range from a sale or financial transaction to an anomaly inferred from data patterns. Data analysis constraints imposed by standard SQL and data warehousing practices, BI reporting and analysis interfaces, and traditional BI dashboards. A gap assessment should include appraisal of current systems ability to respond to business drivers measured against benefits obtainable via adoption of CEP. Resource Inventory A resource inventory will discover neglected or underutilized transactional, data-stream, and event sources clickstreams, credit-card purchases, service invocations, network packets, e-mail and IM traffic, and the like that relate to business drivers. Audit sources and determine their current and unrealized information value. Evaluate access, filtering, and transformation steps required to extract value. Understand how resources will be used in conjunction with data in databases. What outcomes are sought? What interfaces monitoring, interactive analysis, rules system, other are required and how will they close gaps?

Solution Design A solution architecture should respond to both technical and business concerns. These concerns include leveraging existing BI and data warehousing solutions and skill-sets to extend historical analyses and operational monitoring to encompass live data and automated decision making and execution. As part of the design process, it is important to: Include data sources and the analytics necessary to deliver desired outcomes. Account for usability, performance, scalability, reliability, and security. Support required data integration, transformation, and analysis steps; evaluate SOA as an enabling architectural framework. Anticipate new data sources and analytical needs to ensure extendibility. On the business architecture side, it is important to study the effectiveness of current business processes and the potential for realignment. From there, one can see decision-automation possibilities and the best route toward CEP integration into current or reworked business processes in interactive or embedded form.

SOLUTION SAMPLER Complex event processing answers a broad set of business intelligence challenges. The common need is for operational responsiveness, for event-enabled, low-latency analytics. Such solutions go beyond monitoring pre-defined indicators. They deliver ability to act on business events as they occur. For many vendors, operational BI simplistically means running reports off operational systems. They propose a narrow definition of real-time BI as quick (within seconds) response to queries. These limited definitions suit some businesses, but in a world of high transaction volumes, e-business, and sensor networks and monitoring devices given a flood information from financial market, news, Web 2.0 sources these definitions fail to keep up with evolving competitive imperatives. We illustrate three common continuous-intelligence CEP applications. Business Process Control Conventional business activity monitoring (BAM) provides a dashboard interface for real-time display of static performance indicators in graphical and numerical form. While BAM does tap into operational systems, typically within a service-oriented architecture but as frequently via hard-wired connection, conventional systems are rigid and quite limited in scope. CEP extends BAM in two directions. CEP provides an ability to monitor processes by drawing data from multiple sources, detecting interesting and anomalous situations with trace-back to root causes. Second, CEP solutions go beyond monitoring to drive actions in response to business events, helping organizations move from BAM to business process control. Capital Markets Capital markets combine enormous competitive pressures due to the amount of money in play with extreme IT requirements that stem from the volume of data generated and from traders reliance on sophisticated mathematical models that discern opportunities and risks from price quotes, trading data, and company information. Data rates can be extreme, into the hundreds of thousands of market ticks per second, with computations that apply time- and record-windows to current data points, analyzed in conjunction with historical values. Traders look for opportunities that stem from unequal information, and they apply what they find in automated, algorithmic trading. Rapid execution equals competitive edge, an equation that applies in a increasingly set of business domains.

Online Commerce Conventional Web analytics, running off stored Web logs, does not respond to online businesss real-time challenges. Typical reports and charts are not a real-time solution. CEP, by contrast, delivers an ability to act on Web interactions as they are occur, going beyond BI and conventional Web analytics to deliver more responsive, more profitable user experiences. CEP for e-business sees sequences of user actions as a form of data stream. CEPs combination of database queries and event-pattern matching, with continuous computations over time and record windows, can reveal opportunities to better serve site visitors, boost sales conversions, reduce session-abandonment rates, and increase satisfaction and profitability.

GETTING STARTED A basic methodology a series of proven, best-practices steps is an invaluable guide for any organization in the introduction of new technology, Complex Event Processing included. The following is an approach to extending a BI program to encompass stream and messaging sources to support real-time, event-driven analytics. Understand Business Drivers It is essential to justify new technologies and processes in terms of business benefits, whether as a means of pursuing opportunities, mitigating risks, enhancing competitive positioning and profitability, or all of these factors. Several dimensions contribute to ROI: not just money saved or additional revenues earned, but also ability to do more, more effectively. Measures vary by industry. For instance, a sales focused organization will aim to boost customer capture and retention, while a credit-card processor would focus on reducing fraud. What are stakeholders seeking to accomplish. How should achievement be measured? By understanding business drivers, we can address business related objections, which most often focus on cost, mission alignment, and concern about IT capacity and priorities, and win stakeholder buy-in. Address Technical Concerns Study of use cases customer stories that explain how others have harnessed CEP can suggest how solutions can be adapted for an organizations own use. Theyre the basis for evaluation and implementation best practices, and when coupled with proof-of-concept prototyping, can be invaluable in resolving technical concerns: Lack of understanding of streams, event clouds, messaging services, and the business value of the information they contain. Means of addressing short-comings of entrenched approaches & technologies: conventional BI, data warehouses, rigid computing architectures, and so on. The migration from historical-only data query to dynamic, event-driven analysis. Solution performance, robustness, and reliability and the quality of support. A successful POC will demonstrate ability to deliver competitive advantage. It will help an organization create an implementation roadmap that anticipates challenges. Phased Introduction A successful implementation strategy typically involves phased introduction of new technologies, information sources, applications, and business processes. An organization that already has a strong BI/analytics program enjoys a head start. Its nonetheless prudent, when CEP is first being introduced, to target discrete, clearly defined business problems where impact can be measured. The overall solution will involve more than technology. It will account for staff responsibilities, skill-sets, and capacity and include the testing, training, staff augmentation (with outside personnel if needed), and burn-in time necessary to ensure success. A carefully planning CEP introduction and early successes help pave the way for broad acceptance and adoption, turning stakeholders into advocates.

ABOUT THE AUTHOR White paper author Seth Grimes is an information technology analyst and analytics strategy consultant. He is contributing editor at Intelligent Enterprise magazine, founding chair of the Text Analytics Summit, Data Warehousing Institute (TDWI) instructor, and text analytics channel expert at the Business Intelligence Network. Seth founded Washington DC-based Alta Plana Corporation in 1997. He consults, writes, and speaks on information-systems strategy, data management and analysis systems, industry trends, and emerging analytical technologies. Seth can be reached at grimes@altaplana.com, +1 301-270-0795.

Sybase, Inc. Worldwide Headquarters One Sybase Drive Dublin, CA 94568-7902 U.S.A 1 800 8 sybase Copyright 2010 Sybase, an SAP Company. All rights reserved. Unpublished rights reserved under U.S. copyright laws. Sybase and the Sybase logo are trademarks of Sybase, Inc. or its subsidiaries. indicates registration in the United States of America. SAP and the SAP logo are the trademarks or registered trademarks of SAP AG in Germany and in several other countries. All other trademarks are the property of their respective owners. 11/10

www.sybase.com