Big Data Analytics

What, Why and How

Dr.Bharatheesh
Jaysimha
CTO, Abiba Systems
Bengaluru
www.abibasystems.c
om
Data trends
 Personal data
Its coming (2M$ today2K$ in 10 years)
Today the pack rats have ~ 10-100GB l
1-10 GB in text (eMail, PDF, PPT, OCR) n a
s o e
10GB 50GB tiff, mpeg, jpeg, er t )
P b y y
Some have 1TB (voice + video). a d a
h e t e
Video can drive it to 1PB. T P m e
s o
Online PB affordable in 10 years. (
Get ready: tools to capture, manage,
organize, search, display will be big
app.
 Relational Database (SQL)

10 TB to 100TB DBs.
Mostly DSS and data warehouses.
Some media managers
SQL performance is better than
CIFS/NFS
Most bytes are in files
No DBMSs beyond 100TB
 Internet scale data

8,000 no-name PCs

Each 1/3U, 2 x 80 GB disk,
2 cpu 256MB ram
1.4 PB online.
2 TB ram online
8 TeraOps
Slice-price is 1K$ so 8M$.
15 admins (!) (==
1/100TB).
 Unusual Data, but good ones
Particle Physics Hunting the Higgs and
Dark Matter
April 2006: First pp collisions at TeV energies at the
Large Hadron Collider in Geneva
ATLAS/CMS Experiments involve 2000 physicists from
200 organizations in US, EU, Asia
Need to store,access, process, analyse 10 PB/yr with
200 TFlop/s distributed computation
Building hierarchical Grid infrastructure to distribute
data and computation
ExaBytes and PetaFlop/s by 2015
Business Questions
 Business Questions
Which
Whichare
areour
our
lowest/highest
lowest/highestmargin
margin
customers
customers??
Who
Whoare
aremy
mycustomers
customers
What and
andwhat
whatproducts
Whatisisthe
themost
most products
effective are
arethey
theybuying?
effectivedistribution
distribution buying?
channel?
channel?

What
Whatproduct
productprom- Which
prom- Whichcustomers
customers
-otions
-otionshave
havethe
thebiggest are
biggest aremost
mostlikely
likelyto
togo
go
impact
impactononrevenue? to
revenue? tothe
thecompetition
competition??
What
Whatimpact
impactwill
will
new
newproducts/services
products/services
have
haveon
onrevenue
revenue
and
andmargins?
margins?
Current Approach
 EDW and Analytics
Data Sources

Data Warehouse

Data Marts
 Billing System(s)

Daily/Monthly subscriber
change/snapshot records

Canned
Reports
Data Warehouse/Data Marts
Customer Lists
Subscriber Profile Revenue/Usage

Daily Activity Subscriber Cost

Commissions
Customer Contact Subscriber Profitability
Welcome
Letters

Revenue /
Subscriber Revenue
Usage
Activity Plan /
Actuals
Forecast
Data Mining
Actual/plan information
shared between cubes

Relational Queries EIS/Dashboards Excel Templates

Problems with Current Approach
Trends in Analytics
Problems with Current Approach
Size: up to 100 TB
Mostly structured
data
Costly hardware
Costly software

Doubling
almost every
year..
Law of
compounding
!!!
RDBMS

Data load

Size: up to EB
Any data
Commodity hardware
Free software

Hadoopic
Hadoopic
Hadoopic
Hadoopic
 Hadoop
Distributed computing frame work
For clusters of computers
Thousands of Compute Nodes
Petabytes of data
Open source, Java
Googles MapReduce inspired Yahoos
Hadoop.
Now part of Apache group
Hadoop
Hadoop Work flow
Hadoop Sequence diagram
Hadoop mapReduce example
Pig - ETL
 Pig
Started at Yahoo! Research
Features:
Expresses sequences of MapReduce jobs
Data model: nested bags of items
Provides relational (SQL) operators (JOIN,
GROUP BY, etc)
Easy to plug in Java functions
Pig an example of a job

Load Users Load Pages

Filter by age

Join on name

Group on url

Count clicks

Order by clicks

Take top 5
 Pig just look at Java code
importjava.io.IOException;
importjava.util.ArrayList;
importjava.util.Iterator;
importjava.util.List;
importorg.apache.hadoop.fs.Path;
importorg.apache.hadoop.io.LongWritable;
importorg.apache.hadoop.io.Text;
importorg.apache.hadoop.io.Writable;
importorg.apache.hadoop.io.WritableComparable;
importorg.apache.hadoop.mapred.FileInputFormat;
importorg.apache.hadoop.mapred.FileOutputFormat;
importorg.apache.hadoop.mapred.JobConf;
importorg.apache.hadoop.mapred.KeyValueTextInputFormat;
importorg.a pache.hadoop.mapred.Mapper;
importorg.apache.hadoop.mapred.MapReduceBase;
importorg.apache.hadoop.mapred.OutputCollector;
importorg.apache.hadoop.mapred.RecordReader;
importorg.apache.hadoop.mapred.Reducer;
importorg.apache.hadoop.mapred.Reporter;
imp ortorg.apache.hadoop.mapred.SequenceFileInputFormat;
importorg.apache.hadoop.mapred.SequenceFileOutputFormat;
importorg.apache.hadoop.mapred.TextInputFormat;
importorg.apache.hadoop.mapred.jobcontrol.Job;
importorg.apache.hadoop.mapred.jobcontrol.JobC ontrol;
importorg.apache.hadoop.mapred.lib.IdentityMapper;
publicclassMRExample{
publicstaticclassLoadPagesextendsMapReduceBase
implementsMapper<LongWritable,Text,Text,Text>{
publicvoidmap(LongWritablek,Textval,
OutputCollector<Text,Text>oc,
Reporterreporter)throwsIOException{
//Pullthekeyout
Stringline=val.toString();
intfirstComma=line.indexOf(',');
Stringkey=line.sub string(0,firstComma);
Stringvalue=line.substring(firstComma+1);
TextoutKey=newText(key);
//Prependanindextothevaluesoweknowwhichfile
//itcamefrom.
TextoutVal=newText("1 "+value);
oc.collect(outKey,outVal);
} wh ile(iter.hasNext()){
} sum+=iter.next().get();
publicstaticclassLoadAndFilterUsersextendsMapReduceBase
implementsMapper<LongWritable,Text,Text,Text>{
publicvoidmap(LongWritablek,Textval,
OutputCollector<Text,Text>oc,
Reporterreporter)throwsIOException{
//Pullthekeyout
Stringline=val.toString();
intfirstComma=line.indexOf(',');
Stringvalue=line.substring( firstComma+1);
intage=Integer.parseInt(value);
if(age<18||age>25)return;
Stringkey=line.substring(0,firstComma);
TextoutKey=newText(key);
//Prependanindextothevaluesow eknowwhichfile
//itcamefrom.
TextoutVal=newText("2"+value);
oc.collect(outKey,outVal);
} publicstaticclassLimitClicksextendsMapReduceBase
} implementsReducer<LongWritable,Text,LongWritable,Text>{
publicstaticclassJoinextendsMapReduceBase
implementsReducer<Text,Text,Text,Text>{
publicvoidreduce(Textkey,
Iterator<Text>iter,
OutputCollector<Text,Text>oc,
Reporterreporter)throwsIOException{
//Foreachvalue,figureoutwhichfileit'sfromand
storeit
//accordingly.
List<String>first=newArrayList<String>();
List<String>second=newArrayList<String>();
while(iter.hasNext()){
Textt=iter.next();
Stringvalue=t.to String();
if(value.charAt(0)=='1')
first.add(value.substring(1));
elsesecond.add(value.substring(1));
reporter.setStatus("OK"); lp.setOutputKeyClass(Text.class);
} lp.setOutputValueClass(Text.class);
lp.setMapperClass(LoadPages.class);
//Dothecrossproductandcollectthevalues FileInputFormat.addInputPath(lp,new
for(Strings1:first){ Path("/ user/gates/pages"));
for(Strings2:second){ FileOutputFormat.setOutputPath(lp,
Stringoutval=key+","+s1+","+s2; newPath("/user/gates/tmp/indexed_pages"));
oc.collect(null,newText(outval)); lp.setNumReduceTasks(0);
reporter.setStatus("OK"); JobloadPages=newJob(lp);
}
} JobConflfu=newJobConf(MRExample.class);
} lfu.s etJobName("LoadandFilterUsers");
} lfu.setInputFormat(TextInputFormat.class);
publicstaticclassLoadJoinedextendsMapReduceBase lfu.setOutputKeyClass(Text.class);
implementsMapper<Text,Text,Text,LongWritable>{ lfu.setOutputValueClass(Text.class);
lfu.setMapperClass(LoadAndFilterUsers.class);
publicvoidmap( FileInputFormat.add InputPath(lfu,new
Textk, Path("/user/gates/users"));
Textval, FileOutputFormat.setOutputPath(lfu,
OutputColle ctor<Text,LongWritable>oc, newPath("/user/gates/tmp/filtered_users"));
Reporterreporter)throwsIOException{ lfu.setNumReduceTasks(0);
//Findtheurl JobloadUsers=newJob(lfu);
Stringline=val.toString();
intfirstComma=line.indexOf(','); JobConfjoin=newJobConf( MRExample.class);
intsecondComma=line.indexOf(',',first Comma); join.setJobName("JoinUsersandPages");
Stringkey=line.substring(firstComma,secondComma); join.setInputFormat(KeyValueTextInputFormat.class);
//droptherestoftherecord,Idon'tneeditanymore, join.setOutputKeyClass(Text.class);
//justpassa1forthecombiner/reducertosuminstead. join.setOutputValueClass(Text.class);
TextoutKey=newText(key); join.setMapperClass(IdentityMap per.class);
oc.collect(outKey,newLongWritable(1L)); join.setReducerClass(Join.class);
} FileInputFormat.addInputPath(join,new
} Path("/user/gates/tmp/indexed_pages"));
publicstaticclassReduceUrlsextendsMapReduceBase FileInputFormat.addInputPath(join,new
implementsReducer<Text,LongWritable,WritableComparable, Path("/user/gates/tmp/filtered_users"));
Writable>{ FileOutputFormat.se tOutputPath(join,new
Path("/user/gates/tmp/joined"));
publicvoidreduce( join.setNumReduceTasks(50);
Textke y, JobjoinJob=newJob(join);
Iterator<LongWritable>iter, joinJob.addDependingJob(loadPages);
OutputCollector<WritableComparable,Writable>oc, joinJob.addDependingJob(loadUsers);
Reporterreporter)throwsIOException{
//Addupallthevalueswesee JobConfgroup=newJobConf(MRE xample.class);
group.setJobName("GroupURLs");
longsum=0; group.setInputFormat(KeyValueTextInputFormat.class);
group.setOutputKeyClass(Text.class);
group.setOutputValueClass(LongWritable.class);
reporter.setStatus("OK"); group.setOutputFormat(SequenceFi leOutputFormat.class);
} group.setMapperClass(LoadJoined.class);
group.setCombinerClass(ReduceUrls.class);
oc.collect(key,newLongWritable(sum)); group.setReducerClass(ReduceUrls.class);
} FileInputFormat.addInputPath(group,new
} Path("/user/gates/tmp/joined"));
publicstaticclassLoadClicksextendsMapReduceBase FileOutputFormat.setOutputPath(group,new
i mplementsMapper<WritableComparable,Writable,LongWritable, Path("/user/gates/tmp/grouped"));
Text>{ group.setNumReduceTasks(50);
JobgroupJob=newJob(group);
publicvoidmap( groupJob.addDependingJob(joinJob);
WritableComparablekey,
Writableval, JobConftop100=newJobConf(MRExample.class);
OutputCollector<LongWritable,Text>oc, top100.setJobName("Top100sites");
Reporterreporter) throwsIOException{ top100.setInputFormat(SequenceFileInputFormat.class);
oc.collect((LongWritable)val,(Text)key); top100.setOutputKeyClass(LongWritable.class);
} top100.setOutputValueClass(Text.class);
} top100.setOutputFormat(SequenceFileOutputF ormat.class);
top100.setMapperClass(LoadClicks.class);
top100.setCombinerClass(LimitClicks.class);
top100.setReducerClass(LimitClicks.class);
intcount=0; FileInputFormat.addInputPath(top100,new
public voidreduce( Path("/user/gates/tmp/grouped"));
LongWritablekey, FileOutputFormat.setOutputPath(top100,new
Iterator<Text>iter, Path("/user/gates/top100sitesforusers18to25"));
OutputCollector<LongWritable,Text>oc, top100.setNumReduceTasks(1);
Reporterreporter)throwsIOException{ Joblimit=newJob(top100);
limit.addDependingJob(groupJob);
//Onlyoutputthefirst100records
while(count <100&&iter.hasNext()){ JobControljc=newJobControl("Findtop 100sitesforusers
oc.collect(key,iter.next()); 18to25");
count++; jc.addJob(loadPages);
} jc.addJob(loadUsers);
} jc.addJob(joinJob);
} jc.addJob(groupJob);
publicstaticvoidmain(String[]args)throwsIOException{ jc.addJob(limit);
JobConflp=newJobConf(MRExample.class); jc.run();
lp.se tJobName("LoadPages"); }
lp.setInputFormat(TextInputFormat.class); }
 Pig now look at Pig latin
U sers = load users as (nam e, age);
Filtered = filter U sers by
age > = 18 and age < = 25;
Pages = load pages as (user, url);
Joined = join Filtered by nam e, Pages by user;
G rouped = group Joined by url;
Sum m ed = foreach G rouped generate group,
count(Joined) as clicks;
Sorted = order Sum m ed by clicks desc;
Top5 = lim it Sorted 5;

store Top5 into top5sites;

 Pig latin
Notice how naturally the components of the job translate into Pig Latin.

Load Users Load Pages

U sers = load
Filter by age
Filtered = fi
lter
Pages = load
Join on name
Joined = join
Group on url
G rouped = group
Sum m ed = count()
Count clicks Sorted = order
Top5 = lim it
Order by clicks

Take top 5
Hive Data warehouse
 Hive
Developed at Facebook
Used for majority of Facebook jobs
Relational database built on
Hadoop
Maintains list of table schemas
SQL-like query language (HQL)
Can call Hadoop Streaming scripts
from HQL
Supports table partitioning,
clustering, complex data types, some
optimizations
 Hive
Find top 5 pages visited by users aged 18-25:
SELEC T p.url, CO U N T(1) as clicks
FRO M users u JO IN page_view s p O N (u.nam e = p.user)
W H ERE u.age > = 18 AN D u.age < = 25
G RO U P BY p.url
O RD ER BY clicks
LIM IT 5;
Mahout Data mining
Data mining
 Clustering

Perfect candidate for mapReduce

Mahout k-means
Mahout k-means
Mahout k-means
Mahout k-means
Mahout k-means
Mahout Clustering
Big Data Analytics
Big Data Analytics
 Acknowledgements

Thanks to all the authors who left

their slides on the Web.
I own the errors of course.