Bigdata Module1 Topic1 Generalconcepts

Module 1- Fundamentals and Technologies for Big Data Analytics
Big Data, Data Science, and

Data Analytics - Overview
Priv.Doz. Dr. Hong-Linh Truong
truong@rdsea.io
http://truong.rdsea.io
@linhsolar
Mobifone Training. (c) Hong-Linh Truong. Only

11/6/2016
(re)distribution within Mobifone allowed.
1 RDSEA.IO
Who am I?
 Current full time job: Professor at TU Wien – Austria
 http://dsg.tuwien.ac.at/staff/truong/ & http://truong.rdsea.io
 Main function: research and training of PhD and master students
 Before TU Wien: HCMC Uni. Of Technology  Uni. Vienna  Uni. Innsbruck
 Education
 Engineer (Kỹ sư), 1998
 PhD (Tiến sĩ /Tiến sĩ về khoa học), 2005
 Habilitation (Tiến sĩ khoa học), 2013
 Teaching
 In distributed systems, cloud computing, IoT, sevices engineering and big data
 Invited talks and tutorials: IBM, SummerSOC School, CloudCom 2014, Kyoto Uni.,
Shenzhen Uni., Uni. Stuttgart, etc.
 5 PhD graduates, 6 master graduates
 I was a programmer/developer and administrator but not now
11/6/2016
2 RDSEA.IO
for measuring the productivity of individual
Who am I? authors in cloud: rank 24
http://ieeexplore.ieee.org/document/6808484/
 Academic life:
 Publications: ~ 180 papers
 8 best paper awards
 Contribute to acquisition of ~2.5 M EUR for TU Wien (out of ~ 27.5 M
EUR)
 Collaboration Chinese, EU, Vietnam and USA universities
 Industrial experiences and collaboration
 European space Agency (ESA) – 4 projects
 Pacific Controls Cloud Computing Lab: Key member
 Companies (ATOS, Cagemini, Ericsson India, IBM, Ikerlan, DFRC, HP,
Softeco, TXT, etc.), many from through EU projects

11/6/2016
3 RDSEA.IO
Goals and Big data project
manager:
expectation for this Which technologies can I

use for components of
course big data
Computer Scientist:
New Product Goal: the big Which is the most
developer:
picture of big suitable stack I
Is any chance to
should use
develop a new kind of data landscape
applications
and frameworks
Infrastructure
Human Resource
manager:
Manager: who
which kind of
should I hire for the
infrastructures I
big data?
should invest
 Roadmap: the big picture  your focus points  how to work with others  deliver excellent points in
the big picture under your responsibility
 Capture the knowledge  know the state-of-the-art and the trend  assess what others propose to you
 Learning methods: capture the ideas/use cases, discuss the potentials, & do the homework in your fields
11/6/2016
4 RDSEA.IO
What is this lecture about?
The big picture of big data

Big data overview
Data science with big data
Key technical issues
Building the right team

11/6/2016
5 RDSEA.IO
Big Data
 Data: facts, responses,
events, measurement, etc.
{"station_id":"1160629000","datapoint_id":122,"
alarm_id":310,"event_time":"2016-09-
17T02:05:54.000Z","isActive":false,"value":6,"val
ueThreshold":10}
What does it mean

“Big data”?

11/6/2016
6 RDSEA.IO
Is it big?

11/6/2016
7 RDSEA.IO
Why do we have big data now?
 Social media
 data generated by human activities in the Internet
 Facebook, Twitter, Google+, etc.
 Internet of Things (IoT)/Machine-to-Machine (M2M)
 data generated through monitoring devices and environments
 data generated through machines
 Advanced sciences
 data generated by advanced instruments
 earth observation (e.g., Sentinel satellites)
 Personal information (e.g., healthcare)
 Open and transparent government
 open government data
 Etc.
11/6/2016
8 RDSEA.IO
Telecommunication
https://dandelion.eu/datamine/open-big-data/
11/6/2016
9 RDSEA.IO
IoT data in Cities
https://arrayofthings.github.io/node.html

11/6/2016
10 RDSEA.IO
IoT in smart building management
Infrastructure/Internet of Things Internet/public cloud Organization-specific
boundary boundary
Emergency
Management
Equipment Operation and Maintenance
Near Enterprise
Civil protection realtime
analytics Resource
Building Operation Optimization Planning
Pacific controls Predictive
data
analytics Tracking/Log
istics
Visual
Analytics
Infrastructure
Monitoring
...
Cities, e.g. including:

10000+ buildings
1000000+ sensors
11/6/2016
11 RDSEA.IO
Industrial Internet Example: Win turbine analytics
 Monitoring industrial
machines
 Industry 4.0
 IoT and big data
analytics are an
essential part in
manufacturing
processes
Figures source:
http://www.windpowerengineering.com/design/electrical/controls/wind-farm-
networks/talking-turbines-internet-things/

11/6/2016
12 RDSEA.IO
IoT in Healthcare & Insurance
 Healthcare using IoT

 Personal or hospital context
 Insurance business models + big data
 Asset Insurance
 Cars, homes, etc.
 Monitoring + Analytics
 Smart labels
 Asset management and logistics

11/6/2016
13 RDSEA.IO
Smart Agriculture
 Quality of water management
for fish farms
 Land use
Source: Erik Christensen, http://www.sensorfish.eu/
Source: http://www.esa.int

11/6/2016
14 RDSEA.IO
Environmental
monitoring
Source:
http://insights.aclima.io/
Google and Aclima

11/6/2016
15 RDSEA.IO
Earth Observation
Soil moisture
analysis for
Sentinel-1
A lot of input data (L0):

~2.7 TB per day
A lot of results (L1, L2):
e.g., L1 has ~140 MB per
day for a grid of 1kmx1km
Michael Hornacek,Wolfgang Wagner, Daniel Sabel, Hong-Linh Truong, Paul Snoeij, Thomas Hahmann, Erhard Diedrich, Marcela Doubkova, Potential for High Resolution Systematic Global
Surface Soil Moisture Retrieval Via Change Detection Using Sentinel-1, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, April, 2012

11/6/2016
16 RDSEA.IO
Example of open data

11/6/2016
17 RDSEA.IO
Characterize big data
 Often characterized by the concepts of V*: Volume, Variety,

Velocity, Veracity and Valence
 Volume: size (big size, large-data set, massive of small data)
 Variety: complexity (formats, types of data)
 Velocity: speed (generating speed, data movement speed)
 Veracity: quality is very different (bias, accuracy, etc.)
 Valence: relationships among different type of data
 Big challenge for our Vietnamese culture: it is a big problem you

cannot solve it alone (so pls. work with other  )
11/6/2016
18 RDSEA.IO
Discussion: Why do you have to care?
 Do you/your company have/has big data?

 You – not sure -- but your Mobifone – yes !
 Think about big data not just now, but also in the future!
 Where would your Mobifone services go?
 What is your role in the big data landscape?
 Gathering/collecting big data from mobile stations or from call detail record
(CDR), or mobile user service activities?
 Providing platforms for big data? Who are your customers? Mobifone itself?
 Analyzing big data? For which purpose? Reactive monitoring? Predictive
maintaintence? Service recommendation? Marketing?
 Making use of big data for other busineses? Sell data to your customers?

11/6/2016
19 RDSEA.IO
Why do we need to care?
 Because of the values of
data!
 Top-down: Data economy.
 More data  more evidence 
more business success
 Bottom-up
 Optimizing what we have Figure source: McKinsey Global
Institute: THE INTERNET OF
THINGS: MAPPING THE VALUE
 The Unreasonable BEYOND THE HYPE JUNE 2015
Effectiveness of Data” (Alon HIGHLIGHTS
Halevy, Peter Norvig, and

Fernando Pereira)  with
more data, the same
algorithm performs much
more better!
Alon Halevy, Peter Norvig, and Fernando Pereira. 2009. The Unreasonable Effectiveness of Data. IEEE Intelligent Systems 24, 2 (March 2009).
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf

11/6/2016
20 RDSEA.IO
In Telcos - Mobifone
 Network optimizations
 Signal monitoring, usage monitoring
 Bandwidth optimization (e.g., in combination with software-defined networking/NFV)
 Operation analytics and management
 Reactive monitoring
 Predictive maintenance
 Data marketplace and economy
 Selling data about IoT/M2M
 “A 360 degree view” on your customer
 Fraud, service recommendations, customer understanding (churn/segmentation)
 Customer retention, Geo service experience
 New business models and new products
11/6/2016
21 RDSEA.IO
Why do we need to
care?
Not just money but
will you create
impact on the life
of people?

11/6/2016
22 RDSEA.IO
Vietnam‘s application scenarios
http://english.vietnamnet.vn
/fms/vietnam-in-
photos/141752/10-
memorable-photos-of-the-
flooding-in-saigon-on-sept-
15.html
Source: http://english.vietnamnet.vn/fms/society/149411/transport-
minister-vows-to-wipe-out-traffic-congestion-in-vietnam.html
http://www.vacne.org.vn/vietnam-seriously-lacks-water-monitoring-
Source: https://www.theguardian.com
stations/e1414.html

11/6/2016
23 RDSEA.IO
Big data and data science

11/6/2016
24 RDSEA.IO
Big data challenges
 Must handle different types of data from different systems
 V* characteristics
 you need big data infrastructures!
 Must enable both predictive and reactive business analytics and

optimization
 Must understand the semantics of data (it is domain-specific)
Need to develop several analytics (atop of big data infrastructures)
and provision them as services
You need a right team!

11/6/2016
25 RDSEA.IO
Key components in big data
Data Visualization
Data Collection Data Analytics
& Representation
1. Engineering and
operating systems
2. Carrying data
science
3. Making sense of Big Data Infrastructure
data analytics
results
Data Science + Large-scale infrastructure + Service/System Engineering and Provisioning
11/6/2016
26 RDSEA.IO
Data Science
 Key activities in data science So I can just learn

 Exploring and preparing data databases and
(business+technical people)
 Representing, modeling and transforming data data mining?
(technical people) Wrong! you need others: computing
 Analyzing data (technical people + domain systems, domain knowledge, etc
expert + business people)
You need to work with an ecosystem of
 Visualizing and presenting (resulting) data databases and tools
(technical/domain/business people)
George Strawn, "Data Scientist", IT Professional, vol.18, no. 3, pp. 55-57, May-June 2016

11/6/2016
27 RDSEA.IO
Big data infrastructures: the ecosystem
 Stakeholders
 E.g., stakeholders within Mobifone and outside Mobifone
 Data
 Time series machine data, call record details, service usage, signal strength
and bit rate error, etc.
 Infrastructures and systems
 Sensing, communication, storage,
 Services (models, interfaces, rules, etc.)
 Complex event processing, data analytics, optimization, …
 Software & data engineering tooling ecosystem
 Tools and programming language frameworks

11/6/2016
28 RDSEA.IO
Big data infrastructures: Infrastructures and
software systems view
 Distributed data collectors
 For different subsystems as well
 Scalable data storage
 Parallel and distributed processing
 Business services
 Data mining/machine learning algorithms
 Programming environments
 Visualization

11/6/2016
29 RDSEA.IO
Big data infrastructures: tasks
 Sensing and logging

 Communication
 Partitioning data
 Mapping and transforming data
 Processing data
 Scheduling data analytics jobs

11/6/2016
30 RDSEA.IO
Big data infrastructures: processes
 Provisioning and operation processes

 Provide computing resources; managing data sources, backup, etc
 Data processing
 Data pipeline
 Workflows
 Integration with other business services

 Very crucial as well!

11/6/2016
31 RDSEA.IO
Programming models and languages
 Heavily programming tasks

 For provisioning infrastructures and platforms
• Most tasks are software, not hardware (software-defined data centers, networks,
 For programming analytics
 Programming languages, programming models, and
programming environments
 Productivity, reusability and extensibility
• Fast (to meet the market demands)
• Reuse for next projects
• Easy to be extended for new types of data
 Enable diverse types of tasks
11/6/2016
32 RDSEA.IO
Source: http://www.datameer.com/wp-content/uploads/2016/06/matt_turck_big_data_landscape_v11r.png
11/6/2016
33 RDSEA.IO
Quanlity aspects in big data
A lot of
A few platforms
Things
applications
Data/Service Platforms
A lot of
Social Data Ingest Data
data Platforms Storage Applications
...
sources Data Data
Applications
Environments Processing Query
Critical
Data Analytics
Infrastructures
Algorithms/Processes
.... A lot of algorithms and

processes
11/6/2016
34 RDSEA.IO
Takeaway message:
Selecting suitable technologies, design methodologies, and team

is the first condition for
cost-effective solution and high productivity

11/6/2016
35 RDSEA.IO
Big data with key principles in
distributed systems

11/6/2016
36 RDSEA.IO
Big ETL: Extract, Transform and Load
 Traditional ETL: read data from databases (E), convert data,

e.g., using rules (T), and write data to target databases (L)
 Big data ETL
 Big E: Read data from various sources, not just databases but also
logs, instrument, etc. through various protocols
• Different formats and data in motion and data at rest
 Big T: various rules and services to perform transformations, complex
processes for transforming data, batch and realtime data
transformation
 Big L: various types of target sinks (databases, middleware, services,
etc.) using various protocols
11/6/2016
37 RDSEA.IO
Complexity: Not so complex big application
Laptop Technologies Distribution
Web Browser
Internet
Server
Shopping
Recommendation
Server Figure source:

http://drbacchus.com/files/ser
verrack.jpg
Database
(MySQL)
Figure source: https://docs.oracle.com/javaee/7/tutorial/overview003.htm

11/6/2016
38 RDSEA.IO
Complexity: a complex, large-scale big data
processing systems
 Employ several
frameworks and
technologies
 Complex software
stacks
 Large-scale systems
Figure source: http://uidai.gov.in/images/AadhaarTechnologyArchitecture_March2014.pdf

11/6/2016
39 RDSEA.IO
Computing resources: conventional OS, Virtual
Machines, or OS Container?
Source: Kernel-based Virtual Machine
(http://www.linux-kvm.org/page/Main_Page)
Source: The XEN Hypervisor (http://www.xen.org/)

11/6/2016
40 RDSEA.IO
Computing
resources –
clusters, …
Source: http://www.slideshare.net/OpenStack_Online/ibm-
cloud-open-stack-services

11/6/2016
41 RDSEA.IO
Data concerns
data .... Data as a data assets

....
Service
APIs, Querying, Data Management, etc.
Located price?
Quality of data? Privacy
in China?
problem? Service
free? redistribution?
quality?
In Vietnam: it is not a big probem with privacy but things are changing

11/6/2016
42 RDSEA.IO
Partitioning: Splitting functionality & data
 Why?
 Breakdown the complexity
 Easy to implement, replace, and compose
 Deal with performance, scalability, security, etc.
 Support teams in DevOps
 Cope with technology changes

11/6/2016
43 RDSEA.IO
Example of Functional and Data Partitioning
Figures source: http://queue.acm.org/detail.cfm?id=1971597

11/6/2016
44 RDSEA.IO
Coordination: Orchestration and Choreography
Energy Optimization
Service
Sensor Data
Analytics Emergency Service
Orchestration Equipment
Maintenance Service
Near Realtime
Analysis
Sensors Queuing
Historical Data
Choreography Archiving
11/6/2016
45 RDSEA.IO
Distribution: Edge, Network or Data Centers?
 Telcos have great

capabilities to run
part of the big
data game in the
edge!
Figure source: https://portal.etsi.org/portals/0/tbpages/mec/docs/mobile-edge_computing_-_introductory_technical_white_paper_v1%2018-09-14.pdf

11/6/2016
46 RDSEA.IO
Interaction: protocols & interfaces
 Large number of communication protocols and interfaces
 Interaction styles, protocols and interfaces
 REST, SOAP, RPC, Message Passing, Stream-oriented
Communication, Distributed Object models, Component-based
Models
 Your own protocols
 Other criteria
 Architectural constraints
 Scalability, Performance, Adaptability, Monitoring, Logging, etc.

11/6/2016
47 RDSEA.IO
Interaction: Complex interactions
 One-to-many, Many-to-one, Many-to-One

 Message Passping Interface
 Public/Subscribe, Message-oriented Middleware
 Shared Repository
 Application/Systems specific models
Consumer
Client Client Client Client
get delete Producer Middlware Consumer
watch
add
notify Consumer
Repository
Amazon S3
11/6/2016
48 RDSEA.IO
Scalability, Elasticity,
and Availability:
Scale out
Figure source: http://queue.acm.org/detail.cfm?id=2560948

11/6/2016
49 RDSEA.IO
Scalability and Elasticity: Load balancing
Figure source: http://queue.acm.org/detail.cfm?id=1971597

11/6/2016
50 RDSEA.IO
APIs for Monetizing Data
 APIs are key! Why?
 Enable access to data and
Raw
function from entities in your Data
ecosystem without worrying Analytics
about changes within your
organization Notification
 Virtualization Enterprise
....
 An API is an asset
 We need to have lifecycle,
pricing, management, etc.
Check http://www.apiacademy.co for some useful tutorials

11/6/2016
51 RDSEA.IO
Big data and data science –
building a right team

11/6/2016
52 RDSEA.IO
Big data needs a right team
Programming
models, languages,
and methodologies
People in Your Provisioning,

Big data and Operation, analytics
big data team
computing
Platforms Processes

11/6/2016
53 RDSEA.IO
Building a data science team
 System and data engineers:
 building systems and data infrastructures (virtual machines, storage, etc.)
 Data scientists:
 analyze data
 Programmers:
 IT but also data for implementing
 Data science manager
 Business expert
 domain analytics, marketing, etc.
Source/reference: http://radar.oreilly.com/2011/09/building-data-science-teams.html

11/6/2016
54 RDSEA.IO
System and Data Engineer
 Build and maintain Infrastructure

 Computing resources, data storage
 Skills
 Distributed Systems and Computing
 Software systems and services for storage, sensing, procesing, etc
 Tool development
 Software engineering
 Computer Science and Engineering People

11/6/2016
55 RDSEA.IO
Data Scientists
 Analyze data
 Machine learning,
prediction, etc.
 Compute scientist with
domain expertise
 Or domain expert with
compute science skills
Source: http://blog.udacity.com/2014/11/data-science-job-skills.html
11/6/2016
56 RDSEA.IO
Summary
 Big data comes from various sources

 See the value of the data
 Big data cannot be done without big infrastructures
 Rely on many foundations of large-scale distributed systems
 Key issues:
 You need to understand big data ecosystems
 You need to understand the complexity of the underlying distributed systems
 You need to build the right data science team

11/6/2016
57 RDSEA.IO
Further references
 https://www.cloudera.com/content/dam/cloudera/Resources/PDF/sol
ution-briefs/Industry-Brief-Big-Data-Use-Cases-for-Telcos.pdf
 https://aws.amazon.com/blogs/aws/how-the-healthcare-of-tomorrow-
is-being-delivered-today/
 https://azure.microsoft.com/en-us/documentation/articles/data-
science-process-overview/
 http://de.slideshare.net/Hadoop_Summit/monitizing-big-data-at-
telecom-service-providers-33919438
 http://www.forbes.com/sites/bernardmarr/2015/12/16/how-big-data-
is-changing-the-insurance-industry-forever/#5e06093f435e

11/6/2016
58 RDSEA.IO
Thanks for your attention
Hong-Linh Truong
RDSEA.IO
truong@rdsea.io
http://truong.rdsea.io
@linhsolar
Slack: rdsea.slack.com

11/6/2016
59 RDSEA.IO

Bigdata Module1 Topic1 Generalconcepts

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bigdata Module1 Topic1 Generalconcepts

Uploaded by

Copyright:

Available Formats

Module 1- Fundamentals and Technologies for Big Data Analytics

Big Data, Data Science, and

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

expectation for this Which technologies can I

The big picture of big data

Mobifone Training. (c) Hong-Linh Truong. Only

What does it mean

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Cities, e.g. including:

Mobifone Training. (c) Hong-Linh Truong. Only

 Healthcare using IoT

Mobifone Training. (c) Hong-Linh Truong. Only

Source: Erik Christensen, http://www.sensorfish.eu/

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

A lot of input data (L0):

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

 Often characterized by the concepts of V*: Volume, Variety,

 Big challenge for our Vietnamese culture: it is a big problem you

 Do you/your company have/has big data?

Mobifone Training. (c) Hong-Linh Truong. Only

Effectiveness of Data” (Alon HIGHLIGHTS

Halevy, Peter Norvig, and

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

 Must enable both predictive and reactive business analytics and

Mobifone Training. (c) Hong-Linh Truong. Only

 Key activities in data science So I can just learn

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

 Sensing and logging

Mobifone Training. (c) Hong-Linh Truong. Only

 Provisioning and operation processes

 Integration with other business services

Mobifone Training. (c) Hong-Linh Truong. Only

 Heavily programming tasks

.... A lot of algorithms and

Selecting suitable technologies, design methodologies, and team

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

 Traditional ETL: read data from databases (E), convert data,

Server Figure source:

Figure source: https://docs.oracle.com/javaee/7/tutorial/overview003.htm

Mobifone Training. (c) Hong-Linh Truong. Only

Figure source: http://uidai.gov.in/images/AadhaarTechnologyArchitecture_March2014.pdf

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

data .... Data as a data assets

APIs, Querying, Data Management, etc.

Mobifone Training. (c) Hong-Linh Truong. Only

Mobifone Training. (c) Hong-Linh Truong. Only

Figures source: http://queue.acm.org/detail.cfm?id=1971597

Mobifone Training. (c) Hong-Linh Truong. Only

 Telcos have great

Figure source: https://portal.etsi.org/portals/0/tbpages/mec/docs/mobile-edge_computing_-_introductory_technical_white_paper_v1%2018-09-14.pdf