You are on page 1of 4

Complex Event Processing – An Emerging Paradigm in Business Intelligence, Security and Monitoring and Control

Complex Event Processing (CEP) is a technology which has been used for many years in the Aerospace and Defence Industry for Situational Awareness and Data Fusion modules in Command, Control, Communications, Computing and Intelligence Systems (aka C4I). Currently CEP is being rediscovered as a foundation for new class of extremely effective Business Intelligence, Security and System/Network/SCADA Monitoring solutions in industries like Financial Services, Telecommunications, Oil and Gas, Manufacturing, Logistics etc. The increasing connectivity and processing power of the modern IT and Telecom technologies lead to increasing speed and volume of the dataflow available to the organisations. By using CEP solutions companies can gain competitive advantage by achieving real-time situational awareness and tapping the information value that is hidden within the streams of real-time event data that are coming from a variety of sources such as enterprise applications, financial transactions, sensor networks and supply chains. CEP solutions monitor fast moving data streams and leverage the information to achieve operational insight in the areas of Business Intelligence, Security and Monitoring of Systems/Networks/SCADA. CEP can monitor events in real-time, seeking out the patterns and relationships within the data that have meaning to the organization. They can identify important events, event patterns and situations that signal new opportunities, critical threats, changing conditions, or other material factors that will impact the organization. CEP solutions can give organizations increased capacity for competitive action and improve their level of security.

CEP Products
There are several levels of CEP functionality depending on the complexity of event processing and inference. Since CEP (as it is used in business applications) can trace its origins in the Aerospace and Defence Industry it may be appropriate to use the JDL Data Fusion functional model to define the various levels of CEP processing. JDL stands for Joint Directors of Laboratories (JDL) of the US Department of Defence. The JDL functional model specifies 4 levels of Data Fusion functionality with increasing complexity of data processing and inference of high-level information. For the purposes CEP any references to Data Fusion in the JDL model will be replaced with CEP and any references to sensor data and signals with Events. CEP Level 1 (derived from JDL Level 1) - Event pre-processing, refinement and adaptation. Event sorting and correlation into groups with each group representing data related to a higher-level event.

Level 3 and 4 can also be used for automatic control in business process management solutions. There is also an emerging class of CEP products based on techniques such as Bayesian Inference that firmly occupy CEP Level 2 and there is intensive R&D underway to provide the predictive capabilities required at Level 3. StreamBase. Vendors of Rule-Based Inference Products – Progress Apama. they can be used as a standalone CEP engine. Most of the current. CEP Level 4 (derived from JDL Level 4) performs process refinement. In general if the problem can be solved with a list of discrete rules then the problem is not a good candidate for the latter category of CEP products. In business applications of CEP. The power of expression and processing offered by EPL and Pattern Matching Engines is sufficient for CEP Level 1 but not necessarily for complex inference and prediction problems arising at Level 2 and 3. CEP Product Vendors The vendors offering CEP products can be divided into two groups based on the CEP functionality described above. Level 4 interacts with each of the other levels. CEP Level 3 (derived from JDL Level 3) performs predictive analysis of the results from Level 2 to proactively identify threats and opportunities. The rules are written in Event Processing Language (EPL) and then executed by a Pattern Matching Engine directly on the stream of events flowing through it in real-rime. The product from this level is called Situational Awareness. They are based on discrete rules for pattern matching. which is more appropriate for ambiguous and noisy problem domains and CEP Level 2 and 3.CEP Level 2 (derived from JDL Level 2) – Fusing spatial (here spatial doesn’t necessary mean geographical) and temporal relationships between groups of events to infer complex abstract patterns. 2 and 3. CEP products based on techniques such as Bayesian Inference can be used as a complementary CEP engine by organisations requiring CEP solutions covering Level 1. Coral 8 and Esper Vendors of Bayesian Inference Products – Inference Machines CEP Professional Services CEP is still a young industry and consists mainly of product vendors offering . If there is a need for CEP solutions focused mainly on Level 2 and 3. widely spread CEP products offer capabilities corresponding to CEP Level 1 and to some extent Level 2. which is an ongoing monitoring and assessment of the CEP process to refine the process itself and to regulate the acquisition of data to achieve optimal results.

The programmer prepares the Event data for the product and then trains it. Design and Delivery Methodology for Turn-Key CEP Solutions The following is a high-level description of methodology for designing and delivering turn-key CEP solutions. PoC and Training Centre designed from the ground up to test and assess the capabilities of all leading CEP products. Unrivalled experience with large-scale CEP. Design event objects Design event hierarchies and causal models reflecting the target domain Specify the requirements of the CEP project Select CEP product meeting the requirements of the project Instrument the target event sources to produce the required event objects Adapt the CEP engine interfaces to the designed event objects and the organisation’s IT and Telecom infrastructure Develop Pattern Matching Rules in the Event Processing Language (EPL) supported by the selected CEP product This step is valid only for CEP Products based on Bayesian Inference Techniques . Security and Real-Time Enterprise projects in industries like Financial Services. NGN Telecom Solutions etc. refine our CEP Solution Delivery methodologies and train our staff. The trained CEP product can then analyze new Events and act on tem. There are emerging business opportunities for vendor-independent professional services organisations specialised in CEP solutions leveraging key advantages such as: Objectivity when helping clients navigate the CEP industry and advising on product selection. Telecoms. To solve problems from different domains and of different types the product is trained rather than programmed. Real-Time Enterprise.professional services specialised in delivering CEP solutions based on their own CEP products. Energy and Aerospace and Defence. Identify sources of events Identify the low-level event types representing the problem domain. Define the problem – target business domain and complex events and situations of interest. Security. Unmatched experience with both Rule-Based Inference CEP products and CEP Products based on Bayesian Inference Algorithms. Service portfolio which complements CEP with EAI.Design a hierarchy of learning and memory nodes reflecting the problem domain. Use Event Simulation Tools capable of simulating event types and causal models to test the inference capabilities of the CEP solution Use Event Simulation Tools to test the throughput of the CEP solution . iSec Consulting is one example of professional services organisation specialised in turn-key CEP solutions. EAI.

QoS. prediction and detection of system state patterns and failures in complex SCADA systems. security. Automotive Industry . prediction and detection of system state patterns and failures in complex SCADA systems.business intelligence. prediction and detection of system status of complex distributed applications and networks Telecommunications .eftimov@isecc. Aerospace and Defence .business intelligence. Logistics .monitoring.com .situational awareness and data fusion modules for C4I systems. RFID solutions.CEP Solution Scenarios Financial Services .monitoring and detection of system state patterns and failures in complex pipeline systems and power transmission and distribution networks. monitoring. Revenue Assurance. money laundering prevention. algorithmic trading Utilities . Oil and Gas . Author: Evo Eftimov iSec Consulting Ltd www.supply chain management.vehicle management systems (embedded or telematics-based). monitoring.com Mob: +44 (0) 7900847162 Email: evo. prediction and detection of system state patterns in complex networks. security. Energy. fraud prevention.isecc. Manufacturing .plant floor analytics and monitoring. algorithmic trading. RFID solutions.