Real-time Streaming Analytics

Business Value, Use Cases and Architectural Considerations

Big Data Solutions and Services Partner for Enterprises
Anand Venugopal – Sr. Director Business Development Big Data Solutions

1

Yue Cathy Chang – Sr. Director Business Development Alliances and Partnerships

Picture your house

2

What if this was happening now to your home ?

3

Recorded version available at http://bit.ly/1i6OrwR

When do you want to know ?

Later

Now ?
or

4

Recorded version available at http://bit.ly/1i6OrwR

Your best buddy from school

You haven’t met in years
5
Recorded version available at http://bit.ly/1i6OrwR

Is in Vegas same time as you

Your buddy

You

6

Recorded version available at http://bit.ly/1i6OrwR

When do you want to know ?

After you return or
7

NOW ?

Recorded version available at http://bit.ly/1i6OrwR

Life is happening NOW
• Whether individual or Business

• Important things are always happening NOW • NOW is the ONLY time life REALLY happens • Maximize data value  process and act
sooner!

Real-time insight preserves or creates value
8

Topics we will cover today
What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?
Use cases

How?
Architectural considerations

What next ?
Recommendations
9

Recorded version available at http://bit.ly/1i6OrwR

What is Real-time Streaming Analytics ?
What it is NOT
• Quick-response interactive analytics on static data
• Batch processing • Could be close but still NOT – Micro Batch processing

10

Recorded version available at http://bit.ly/1i6OrwR

What is Real-time Streaming Analytics ?
• Data analyzed in motion – as it arrives

• Routine: Monitoring, Counting, Alerting, Reporting
• Complex Decision Making with Predictive analytics • Every incoming event is distinctly processible • Receive, Inspect, Analyse, Store, Distribute • Events may be stored later or in parallel • Immediate actions possible after processing

11

Recorded version available at http://bit.ly/1i6OrwR

Real time vs. Batch analytics
BATCH
SENSE Days ANALYSE Week s ACT

Sec/ ms

Sec/ ms

Real time

SENSE

ANALYSE

ACT

12

Recorded version available at http://bit.ly/1i6OrwR

13

Topics we will cover today
What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?
Use cases

How?
Architectural considerations

What next ?
Recommendations
14

Recorded version available at http://bit.ly/1i6OrwR

Business Value
Diminishes with the age of data

$$$ ? Befor e

• Predictive analytics based on current events • Value depends on accuracy • Real-time • Certainty is high – REAL • Value based on quick response

$$ NOW

$  $ $ Later
Value of Data
Age of Data

• Descriptiv e • Diagnostic • Least value

The drop is non-linear
15

Business Value from RTSA
• Routine business operations (Real time systems)

• Cutting preventable losses
• Finding and monetizing missed opportunities
• More revenue

• Cost savings

• Creating new opportunities
• New Business models (Products, Services, Revenue)

16

Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today
What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?
Use cases

How?
Architectural considerations

What next ?
Recommendations
17

Recorded version available at http://bit.ly/1i6OrwR

Routine Operations
(RT systems)
• Manufacturing – Control Systems (Closed loop)

• IT - Systems & Network Monitoring
• Field Assets Monitoring and Alerting • Trucks, Oil rigs, Vending machines, Radio towers

• Financial Transactions Processing
• Authentications, Validations, Fraud

18

Recorded version available at http://bit.ly/1i6OrwR

Cutting Preventable Losses
Many headline stories are failures in routine operations and were preventable losses

• MH 370 – Loss of lives and assets • GM – Manufacturing defects • Target – Major Security breach • Stock Exchange Meltdown

19

Recorded version available at http://bit.ly/1i6OrwR

Cutting Preventable Losses (2)
• Medical / Clinical – Complex analytics in ICU

• Disaster Warning Systems: Chile / Sandy
• Brokerage - Fraudulent or Risky Trades • Preventive Maintenance – Machines, Plants

• Customer Churn
• Brand Reputation on Social Media

20

Recorded version available at http://bit.ly/1i6OrwR

Missed Opportunities - Revenue
• Customer Service always happens in real-time

• Listening and Learning from customers (Social)
• Context sensitive inventory – Products, Ads • Recommend - Upsell – Cross-sell

SENSE Arrival

ANALYSE Context

ACT Serve

21

Recorded version available at http://bit.ly/1i6OrwR

Missed Opportunities - Efficiency
• Operational Efficiency of systems or processes • Network Optimization for cost, quality of service • Dynamic capacity management • Dynamic re-routing of traffic, cargo • Insurance Adjudication – Drone image analysis
SENSE ANALYSE ACT

Operation

Efficiency

Change

22

Recorded version available at http://bit.ly/1i6OrwR

New Opportunities
• Tractors are becoming soil sensors • Information service to farmers

• Nike – becoming a healthcare company ??
• Quantified self movement • Telecom giants selling data and insights

SENSE What else ?

ANALYSE What else ?

ACT What else ?

23

Recorded version available at http://bit.ly/1i6OrwR

Business Value of RTSA Summary

24

Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today
What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?
Use cases

How ?
Architectural considerations

What next ?
Recommendations
25

Recorded version available at http://bit.ly/1i6OrwR

Architectural Considerations
(1/3)

SENSE SENSE

ACT ACT

ANALYSE ANALYSE

APPROPRIATELY

26

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics
Batch Processin g

Stream Processi ng Real-time Analytics

27

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

Stream Processing

• Reliable ingest of multiple hi-volume hi-speed streams • Route Data to Indexing and/or Storing DBs • Distribution of data and analytics results • Dynamic rule based routing

28

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

• Hi-speed ingest of multiple hivolume streams • Route to Indexing and Storing • Distribution of data and analytics results over ESB • Dynamic rule based routing

Stream Processin g

Real-time Analytics

• Computations • Rules-based Monitoring & Alerts • Predictive Models – live scoring of real time data with run-time model refresh

29

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics
• Receive, store and “active archive” of data-feeds • Offline analytics over ALL data • Visual interactive BI/ Dashboards Batch • Offline building and updating of Predictive Processin Models for real-time scoring

g

• Hi-speed ingest of multiple hi-volume streams • Route to Index and DB • Distribution of data and analytics results over ESB • Dynamic rule based routing

Stream Processin g Real-time Analytics
• Computations • Rules-based Monitoring & Alerts • Predictive Models – live scoring of real time data with run-time model refresh

30

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics
• Receive, store and “active archive” of data-feeds • Offline analytics over ALL data • Visual interactive BI/ Dashboards Batch • Offline building and updating of Processing Predictive Models for real-time scoring

Stream Processin g
• Hi-speed ingest of multiple hivolume streams • Route to Index and DB • Distribution of data and analytics results over ESB • Dynamic rule based routing

Real-time Analytics
• Computations • Rules-based Monitoring & Alerts • Predictive Models – live scoring of real time data with run-time model refresh

31

Recorded version available at http://bit.ly/1i6OrwR

Real-time streaming analytics pipeline and flow

32

Recorded version available at http://bit.ly/1i6OrwR

Real-time streaming analytics pipeline and flow

Scale and Robustness Reliability Guarantees Publish-Subscribe Flexibility – Dynamic Integration with Batch Loose Coupling Visualization Ease of Administration
33

Recorded version available at http://bit.ly/1i6OrwR

StreamAnalytix

34

Recorded version available at http://bit.ly/1i6OrwR

Approaches to Stream Analytics
• Proprietary platforms • Vendor lock-in • No leverage of open source movement • Do it yourself

• Open source stitch up
• Integration and maintenance nightmare • Significant delays in time-to-market

35

Recorded version available at http://bit.ly/1i6OrwR

Introducing ‘StreamAnalytix’
• An “App Server” for real-time apps • Based on best-of-breed Open source • Focus on your Business logic leave infrastructure to the
platform

• Handle all the 3V’s of Big Data in one platform • Seamless integration with Hadoop, NoSQL or any other DB • Rapidly operationalize pre-built analytical models or new ones • Significant time to market acceleration

• Impetus provides full product support and professional
36

services

Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today
What ?
Is Real time streaming anaytics and what it is not

Why ?
Business Value

Who and Where ?
Use cases

How?
Architectural considerations

What next ?
Recommendations
37

Recorded version available at http://bit.ly/1i6OrwR

RECAP
• Important things are always happening NOW

• Maximize data value  process and act sooner!
• There is value – find it  Improve Ops, Cut losses, Find
missed & new opportunities

• Architecture: Sense  Analyse  Act  Sense

Real time insight preserves and creates business value
38 38

RECOMMENDATIONS
• Get Real time streaming analytics in your roadmap

• Talk to experienced peers and consultants
• Start now with opportunities search, solution architecture and
vendor conversations

• Instrument (SENSE) everything – find gaps and fill
• Prove value with “faster batch” with current infra is possible • Establish mechanisms to ACT on the insights

• Close the loop – Sense and Analyse effectiveness
• DO IT
39
Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today
What ?
Is Real time streaming anaytics and what it is not

Why ?
Business Value

Who and Where ?
Use cases

How?
Architectural considerations

What next ?
Recommendations
40

Request a demo of StreamAnalytix
bigdata@impetus.com

Big Data Solutions and Services partner for Enterprises
@impetustec h

41

Recorded version available at http://bit.ly/1i6OrwR