Pig Program - Odt

rr@ubuntu:~$ start-all.
sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/09/24 14:25:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-rr-namenode-ubuntu.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-rr-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-rr-secondarynamenode-
ubuntu.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-rr-resourcemanager-ubuntu.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-rr-nodemanager-ubuntu.out
rr@ubuntu:~$ jps
3536 SecondaryNameNode
3249 NameNode
4099 Jps
3371 DataNode
3757 ResourceManager
3887 NodeManager
rr@ubuntu:~$ hdfs dfs -ls
Found 3 items
drwxr-xr-x - rr supergroup 0 2019-07-23 14:37 dir11
drwxr-xr-x - rr supergroup 0 2019-07-23 15:13 gayathri
drwxr-xr-x - rr supergroup 0 2019-07-30 15:21 testfile.txt
rr@ubuntu:~$ hdfs dfs -cat testfile.txt
cat: `testfile.txt': Is a directory
rr@ubuntu:~$ hdfs dfs -ls /dir11
Found 1 items
-rwx------ 1 rr supergroup 117 2019-03-05 15:05 /dir11/sample2.txt
rr@ubuntu:~$ hdfs dfs -cat /dir11/sample2.txt
it does not support writing of data into the files using multiple writers.the best for reading data either
from end.
rr@ubuntu:~$ gedit emp.txt
rr@ubuntu:~$ hdfs dfs -put empx.txt hdfs://localhost:54310/dir11
put: `empx.txt': No such file or directory
rr@ubuntu:~$ hdfs dfs -put emp.txt hdfs://localhost:54310/dir11
rr@ubuntu:~$ hdfs dfs -ls /dir11
Found 2 items
-rw-r--r-- 1 rr supergroup 43 2019-09-24 14:33 /dir11/emp.txt
-rwx------ 1 rr supergroup 117 2019-03-05 15:05 /dir11/sample2.txt
rr@ubuntu:~$
rr@ubuntu:~$
rr@ubuntu:~$ pig -x local
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-
1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/09/24 14:35:30 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
19/09/24 14:35:30 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2019-09-24 14:35:30,823 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0 (r1746530)
compiled Jun 01 2016, 23:10:49
2019-09-24 14:35:30,830 [main] INFO org.apache.pig.Main - Logging error messages to:
/home/rr/pig_1569364530821.log
2019-09-24 14:35:30,956 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file
/home/rr/.pigbootup not found
2019-09-24 14:35:32,706 [main] INFO org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
fs.default.name is deprecated. Instead, use fs.defaultFS
2019-09-24 14:35:32,755 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file
system at: file:///
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2019-09-24 14:35:33,587 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-
default-865c4ad5-0f14-4317-bd8f-79b37a45ccd6
2019-09-24 14:35:33,594 [main] WARN org.apache.pig.PigServer - ATS is disabled since
yarn.timeline-service.enabled set to false
grunt> a1=LOAD 'hdfs://localhost:54310/dir11/emp.txt USING PigStorage(',') AS
(id:int,name:chararray);
2019-09-24 14:37:21,392 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error
during parsing. Encountered " <PATH> "a1=LOAD "" at line 1, column 1.
Was expecting one of:
<EOF>
"cat" ...
"clear" ...
"fs" ...
"sh" ...
"cd" ...
"cp" ...
"copyFromLocal" ...
"copyToLocal" ...
"dump" ...
"\\d" ...
"describe" ...
"\\de" ...
"aliases" ...
"explain" ...
"\\e" ...
"help" ...
"history" ...
"kill" ...
"ls" ...
"mv" ...
"mkdir" ...
"pwd" ...
"quit" ...
"\\q" ...
"register" ...
"rm" ...
"rmf" ...
"set" ...
"illustrate" ...
"\\i" ...
"run" ...
"exec" ...
"%default" ...
"%declare" ...
"scriptDone" ...
"" ...
"" ...
<EOL> ...
";" ...
Details at logfile: /home/rr/pig_1569364530821.log

grunt> a1=LOAD 'hdfs://localhost:54310/dir11/emp.txt' USING PigStorage(',') AS
2019-09-24 14:37:54,452 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error
during parsing. Encountered " <PATH> "a1=LOAD "" at line 1, column 1.
Was expecting one of:
<EOF>
"cat" ...
"clear" ...
"fs" ...
"sh" ...
"cd" ...
"cp" ...
"copyFromLocal" ...
"copyToLocal" ...
"dump" ...
"\\d" ...
"describe" ...
"\\de" ...
"aliases" ...
"explain" ...
"\\e" ...
"help" ...
"history" ...
"kill" ...
"ls" ...
"mv" ...
"mkdir" ...
"pwd" ...
"quit" ...
"\\q" ...
"register" ...
"rm" ...
"rmf" ...
"set" ...
"illustrate" ...
"\\i" ...
"run" ...
"exec" ...
"%default" ...
"%declare" ...
"scriptDone" ...
"" ...
"" ...
<EOL> ...
";" ...
Details at logfile: /home/rr/pig_1569364530821.log

grunt> A1 = LOAD 'hdfs://localhost:54310/dir11/emp.txt' USING PigStorage(',') AS
2019-09-24 14:41:15,048 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where applicable
grunt> a1 = LOAD 'hdfs://localhost:54310/dir11/emp.txt USING PigStorage(',') AS
>> dump A1;
>> dump a1;
>> A1 = LOAD 'hdfs://localhost:54310/dir11/emp.txt' USING PigStorage(',') AS
>> pig -x local
>>
>>
>>
>>
>>
>>
>>
[1]+ Stopped pig -x local
rr@ubuntu:~$ ^Zz;
#z: command not found
rr@ubuntu:~$
rr@ubuntu:~$ pig -x local
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-
SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/09/24 14:45:54 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
19/09/24 14:45:54 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2019-09-24 14:45:54,472 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0 (r1746530)
compiled Jun 01 2016, 23:10:49
2019-09-24 14:45:54,472 [main] INFO org.apache.pig.Main - Logging error messages to:
/home/rr/pig_1569365154470.log
2019-09-24 14:45:54,592 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file
/home/rr/.pigbootup not found
2019-09-24 14:45:57,357 [main] INFO
system at: file:///
2019-09-24 14:45:58,001 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-
default-ad7a727d-02ac-4f9d-8759-2b573fecc5e2
2019-09-24 14:45:58,002 [main] WARN org.apache.pig.PigServer - ATS is disabled since
yarn.timeline-service.enabled set to false
grunt> D1 = LOAD 'hdfs://localhost:54310/dir11/emp.txt' USING PigStorage(',') AS
2019-09-24 14:48:43,809 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where applicable
grunt> dump D1;
2019-09-24 14:48:59,722 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in
the script: UNKNOWN
2019-09-24 14:49:00,229 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach,
ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer,
LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer,
PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter,
StreamTypeCastInserter]}
2019-09-24 14:49:01,331 [main] INFO org.apache.pig.impl.util.SpillableMemoryManager - Selected
heap (Tenured Gen) of size 699072512 to monitor. collectionUsageThreshold = 489350752,
usageThreshold = 489350752
2019-09-24 14:49:01,541 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation
threshold: 100 optimistic? false
2019-09-24 14:49:01,610 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan
size before optimization: 1
2019-09-24 14:49:01,612 [main] INFO
size after optimization: 1
2019-09-24 14:49:03,736 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - session.id
is deprecated. Instead, use dfs.metrics.session-id
2019-09-24 14:49:04,364 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM
Metrics with processName=JobTracker, sessionId=
2019-09-24 14:49:04,712 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig
script settings are added to the job
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2019-09-24 14:49:04,726 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler -
mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2019-09-24 14:49:04,880 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up
single store job
2019-09-24 14:49:04,962 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key
[pig.schematuple] is false, will not generate code.
2019-09-24 14:49:04,966 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process
to move generated code to distributed cacche
2019-09-24 14:49:04,966 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Distributed cache
not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp
directory: /tmp/1569365344962-0
2019-09-24 14:49:05,283 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-
reduce job(s) waiting for submission.
mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
2019-09-24 14:49:05,391 [JobControl] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2019-09-24 14:49:05,674 [JobControl] INFO org.apache.hadoop.conf.Configuration.deprecation -
mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2019-09-24 14:49:05,858 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader -
No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2019-09-24 14:49:05,961 [JobControl] INFO org.apache.pig.builtin.PigStorage - Using
PigTextInputFormat
2019-09-24 14:49:06,007 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
- Total input paths to process : 1
2019-09-24 14:49:06,011 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2019-09-24 14:49:06,373 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to
process : 1
2019-09-24 14:49:06,596 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of
splits:1
2019-09-24 14:49:07,471 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter -
Submitting tokens for job: job_local1655394155_0001
2019-09-24 14:49:08,230 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the
job: http://localhost:8080/
2019-09-24 14:49:08,233 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
HadoopJobId: job_local1655394155_0001
2019-09-24 14:49:08,233 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing
aliases D1
2019-09-24 14:49:08,234 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed
locations: M: D1[1,5],D1[-1,-1] C: R:
2019-09-24 14:49:08,260 [Thread-20] INFO org.apache.hadoop.mapred.LocalJobRunner -
OutputCommitter set in config null
2019-09-24 14:49:08,261 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0%
complete
2019-09-24 14:49:08,261 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running
jobs are [job_local1655394155_0001]
2019-09-24 14:49:08,453 [Thread-20] INFO org.apache.hadoop.conf.Configuration.deprecation -
2019-09-24 14:49:08,469 [Thread-20] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - File Output Committer Algorithm
version is 1
OutputCommitter is
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter
2019-09-24 14:49:08,738 [Thread-20] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for
map tasks
2019-09-24 14:49:08,739 [LocalJobRunner Map Task Executor #0] INFO
org.apache.hadoop.mapred.LocalJobRunner - Starting task:
attempt_local1655394155_0001_m_000000_0
version is 1
org.apache.hadoop.mapred.Task - Using ResourceCalculatorProcessTree : [ ]
org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 43
Input split[0]:
Length = 43
ClassName: org.apache.hadoop.mapreduce.lib.input.FileSplit
Locations:
-----------------------

org.apache.pig.builtin.PigStorage - Using PigTextInputFormat
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split
being processed hdfs://localhost:54310/dir11/emp.txt:0+43
version is 1
org.apache.pig.impl.util.SpillableMemoryManager - Selected heap (Tenured Gen) of size 699072512 to
monitor. collectionUsageThreshold = 489350752, usageThreshold = 489350752
org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate
code.
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being
processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1] C: R:
org.apache.hadoop.mapred.LocalJobRunner -
org.apache.hadoop.mapred.Task - Task:attempt_local1655394155_0001_m_000000_0 is done. And is
in the process of committing
org.apache.hadoop.mapred.Task - Task attempt_local1655394155_0001_m_000000_0 is allowed to
commit now
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task
'attempt_local1655394155_0001_m_000000_0' to
file:/tmp/temp1989380716/tmp1997065368/_temporary/0/task_local1655394155_0001_m_000000
org.apache.hadoop.mapred.LocalJobRunner - map
org.apache.hadoop.mapred.Task - Task 'attempt_local1655394155_0001_m_000000_0' done.
org.apache.hadoop.mapred.LocalJobRunner - Finishing task:
attempt_local1655394155_0001_m_000000_0
2019-09-24 14:49:10,397 [Thread-20] INFO org.apache.hadoop.mapred.LocalJobRunner - map task
executor complete.
2019-09-24 14:49:10,636 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize
JVM Metrics with processName=JobTracker, sessionId= - already initialized
mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2019-09-24 14:49:10,788 [main] INFO
complete
2019-09-24 14:49:10,790 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats -
Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features

2.7.1 0.16.0 rr 2019-09-24 14:49:04 2019-09-24 14:49:10 UNKNOWN
Success!
Job Stats (time in seconds):

JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime
MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias
Feature Outputs
job_local1655394155_0001 1 0 n/a n/a n/a n/a 0 0 0 0
D1MAP_ONLY file:/tmp/temp1989380716/tmp1997065368,
Input(s):
Successfully read 4 records (43 bytes) from: "hdfs://localhost:54310/dir11/emp.txt"
Output(s):
Successfully stored 4 records in: "file:/tmp/temp1989380716/tmp1997065368"
Counters:
Total records written : 4
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_local1655394155_0001

2019-09-24 14:49:10,819 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2019-09-24 14:49:10,838 [main] WARN org.apache.pig.data.SchemaTupleBackend -
SchemaTupleBackend has already been initialized
2019-09-24 14:49:10,930 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat -
Total input paths to process : 1
2019-09-24 14:49:10,932 [main] INFO
(12, Narien)
(14, Nandhu)
(16, Naveen)
(18, Nagul)
grunt> idlimit = FILTER D1 BY id>14;
grunt> dump idlimit;
the script: FILTER
2019-09-24 14:50:27,387 [main] INFO
2019-09-24 14:50:27,393 [main] INFO
2019-09-24 14:50:27,399 [main] INFO
2019-09-24 14:50:27,401 [main] INFO
2019-09-24 14:50:27,497 [main] INFO
2019-09-24 14:50:27,504 [main] INFO
single store job
directory: /tmp/1569365427504-0
2019-09-24 14:50:27,587 [main] INFO
PigTextInputFormat
2019-09-24 14:50:27,727 [JobControl] INFO
2019-09-24 14:50:27,732 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:50:28,387 [main] INFO
2019-09-24 14:50:28,387 [main] INFO
aliases D1,idlimit
2019-09-24 14:50:28,387 [main] INFO
locations: M: D1[1,5],D1[-1,-1],idlimit[2,10] C: R:
2019-09-24 14:50:28,389 [main] INFO
complete
2019-09-24 14:50:28,389 [main] INFO
2019-09-24 14:50:28,439 [Thread-45] INFO
version is 1
OutputCommitter is
map tasks
attempt_local1084196401_0002_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

version is 1
code.
processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1],idlimit[2,10] C: R:
commit now
'attempt_local1084196401_0002_m_000000_0' to file:/tmp/temp1989380716/tmp-
2053029601/_temporary/0/task_local1084196401_0002_m_000000
attempt_local1084196401_0002_m_000000_0
executor complete.
2019-09-24 14:50:28,902 [main] INFO
complete
Script Statistics:

2.7.1 0.16.0 rr 2019-09-24 14:50:27 2019-09-24 14:50:28 FILTER
Success!

Feature Outputs
D1,idlimit MAP_ONLY file:/tmp/temp1989380716/tmp-2053029601,
Input(s):
Output(s):
Successfully stored 2 records in: "file:/tmp/temp1989380716/tmp-2053029601"
Counters:
Job DAG:
job_local1084196401_0002

2019-09-24 14:50:28,967 [main] INFO
2019-09-24 14:50:28,999 [main] INFO
(16, Naveen)
(18, Nagul)
grunt> describe D1;
D1: {id: int,name: chararray}
grunt> groupbyid = GROUP D1 BY id;
grunt> dump groupbyid;
the script: GROUP_BY
2019-09-24 14:52:55,634 [main] INFO
2019-09-24 14:52:55,647 [main] INFO
2019-09-24 14:52:55,659 [main] INFO
2019-09-24 14:52:55,674 [main] INFO
2019-09-24 14:52:55,714 [main] INFO
2019-09-24 14:52:55,715 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase
detected, estimating # of required reducers.
2019-09-24 14:52:55,727 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using
reducer estimator:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2019-09-24 14:52:55,733 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator -
BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=43
2019-09-24 14:52:55,733 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting
Parallelism to 1
2019-09-24 14:52:55,739 [main] INFO
single store job
directory: /tmp/1569365575741-0
2019-09-24 14:52:55,934 [main] INFO
PigTextInputFormat
2019-09-24 14:52:56,120 [JobControl] INFO
2019-09-24 14:52:56,123 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:52:56,692 [main] INFO
2019-09-24 14:52:56,693 [main] INFO
aliases D1,groupbyid
2019-09-24 14:52:56,693 [main] INFO
locations: M: D1[1,5],D1[-1,-1],groupbyid[3,12] C: R:
2019-09-24 14:52:56,697 [main] INFO
complete
2019-09-24 14:52:56,697 [main] INFO
2019-09-24 14:52:56,706 [Thread-68] INFO
version is 1
OutputCommitter is
map tasks
attempt_local231953609_0003_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

org.apache.hadoop.mapred.MapTask - (EQUATOR) 0 kvi 26214396(104857584)
org.apache.hadoop.mapred.MapTask - mapreduce.task.io.sort.mb: 100
org.apache.hadoop.mapred.MapTask - soft limit at 83886080
org.apache.hadoop.mapred.MapTask - bufstart = 0; bufvoid = 104857600
org.apache.hadoop.mapred.MapTask - kvstart = 26214396; length = 6553600
org.apache.hadoop.mapred.MapTask - Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
code.
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map -
Aliases being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1],groupbyid[3,12]
C: R:
org.apache.hadoop.mapred.MapTask - Starting flush of map output
org.apache.hadoop.mapred.MapTask - Spilling map output
org.apache.hadoop.mapred.MapTask - bufstart = 0; bufend = 75; bufvoid = 104857600
org.apache.hadoop.mapred.MapTask - kvstart = 26214396(104857584); kvend =
26214384(104857536); length = 13/6553600
org.apache.hadoop.mapred.MapTask - Finished spill 0
org.apache.hadoop.mapred.Task - Task:attempt_local231953609_0003_m_000000_0 is done. And is in
the process of committing
attempt_local231953609_0003_m_000000_0
executor complete.
reduce tasks
2019-09-24 14:53:01,310 [pool-10-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner -
Starting task: attempt_local231953609_0003_r_000000_0
2019-09-24 14:53:01,347 [pool-10-thread-1] INFO
version is 1
2019-09-24 14:53:01,368 [pool-10-thread-1] INFO org.apache.hadoop.mapred.Task - Using
ResourceCalculatorProcessTree : [ ]
2019-09-24 14:53:01,390 [pool-10-thread-1] INFO org.apache.hadoop.mapred.ReduceTask - Using
ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@703e83d7
2019-09-24 14:53:01,415 [main] INFO
complete
2019-09-24 14:53:01,415 [main] INFO
2019-09-24 14:53:01,486 [pool-10-thread-1] INFO
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - MergerManager:
memoryLimit=709551680, maxSingleShuffleLimit=177387920, mergeThreshold=468304128,
ioSortFactor=10, memToMemMergeOutputsThreshold=10
2019-09-24 14:53:01,550 [EventFetcher for fetching Map Completion Events] INFO
org.apache.hadoop.mapreduce.task.reduce.EventFetcher - attempt_local231953609_0003_r_000000_0
Thread started: EventFetcher for fetching Map Completion Events
2019-09-24 14:53:01,721 [localfetcher#1] INFO
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - localfetcher#1 about to shuffle output of map
attempt_local231953609_0003_m_000000_0 decomp: 85 len: 89 to MEMORY
org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput - Read 85 bytes from map-output for
attempt_local231953609_0003_m_000000_0
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - closeInMemoryFile -> map-output of
size: 85, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->85
org.apache.hadoop.mapreduce.task.reduce.EventFetcher - EventFetcher is interrupted.. Returning
2019-09-24 14:53:01,806 [pool-10-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1
copied.
2019-09-24 14:53:01,806 [pool-10-thread-1] INFO
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - finalMerge called with 1 in-memory
map-outputs and 0 on-disk map-outputs
2019-09-24 14:53:01,857 [pool-10-thread-1] INFO org.apache.hadoop.mapred.Merger - Merging 1
sorted segments
2019-09-24 14:53:01,861 [pool-10-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the
last merge-pass, with 1 segments left of total size: 77 bytes
2019-09-24 14:53:01,865 [pool-10-thread-1] INFO
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merged 1 segments, 85 bytes to disk
to satisfy reduce memory limit
2019-09-24 14:53:01,867 [pool-10-thread-1] INFO
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 1 files, 89 bytes from disk
2019-09-24 14:53:01,868 [pool-10-thread-1] INFO
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 0 segments, 0 bytes from
memory into reduce
sorted segments
copied.
2019-09-24 14:53:01,933 [pool-10-thread-1] INFO
version is 1
2019-09-24 14:53:02,099 [pool-10-thread-1] INFO org.apache.hadoop.conf.Configuration.deprecation
- mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2019-09-24 14:53:02,142 [pool-10-thread-1] INFO org.apache.pig.impl.util.SpillableMemoryManager
- Selected heap (Tenured Gen) of size 699072512 to monitor. collectionUsageThreshold = 489350752,
2019-09-24 14:53:02,143 [pool-10-thread-1] WARN org.apache.pig.data.SchemaTupleBackend -
2019-09-24 14:53:02,146 [pool-10-thread-1] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - Aliases
being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1],groupbyid[3,12] C: R:
2019-09-24 14:53:02,952 [pool-10-thread-1] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local231953609_0003_r_000000_0 is done. And is in the process of committing
copied.
2019-09-24 14:53:02,975 [pool-10-thread-1] INFO org.apache.hadoop.mapred.Task - Task
attempt_local231953609_0003_r_000000_0 is allowed to commit now
2019-09-24 14:53:03,017 [pool-10-thread-1] INFO
'attempt_local231953609_0003_r_000000_0' to file:/tmp/temp1989380716/tmp-
123884449/_temporary/0/task_local231953609_0003_r_000000
reduce > reduce
'attempt_local231953609_0003_r_000000_0' done.
Finishing task: attempt_local231953609_0003_r_000000_0
2019-09-24 14:53:03,030 [Thread-68] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce task
executor complete.
2019-09-24 14:53:03,353 [main] INFO
complete
Script Statistics:

2.7.1 0.16.0 rr 2019-09-24 14:52:55 2019-09-24 14:53:03 GROUP_BY
Success!

Feature Outputs
job_local231953609_0003 1 1 n/a n/a n/a n/a n/a n/a n/a n/a
D1,groupbyid GROUP_BY file:/tmp/temp1989380716/tmp-123884449,
Input(s):
Output(s):
Counters:
Job DAG:
job_local231953609_0003

2019-09-24 14:53:03,402 [main] INFO
2019-09-24 14:53:03,595 [main] INFO
(12,{(12, Narien)})
(14,{(14, Nandhu)})
(16,{(16, Naveen)})
(18,{(18, Nagul)})
grunt> maxid = FOREACH groupbyid GENERATE group,COUNT(D1.id);
grunt> dump maxid;
2019-09-24 14:54:55,081 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key
[pig.schematuple] was not set... will not generate code.
2019-09-24 14:54:55,081 [main] INFO
2019-09-24 14:54:55,127 [main] INFO
2019-09-24 14:54:55,129 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move
algebraic foreach to combiner
2019-09-24 14:54:55,162 [main] INFO
2019-09-24 14:54:55,183 [main] INFO
2019-09-24 14:54:55,278 [main] INFO
2019-09-24 14:54:55,278 [main] INFO
2019-09-24 14:54:55,278 [main] INFO
reducer estimator:
2019-09-24 14:54:55,281 [main] INFO
2019-09-24 14:54:55,281 [main] INFO
Parallelism to 1
2019-09-24 14:54:55,283 [main] INFO
single store job
directory: /tmp/1569365695286-0
2019-09-24 14:54:55,431 [main] INFO
PigTextInputFormat
2019-09-24 14:54:55,498 [JobControl] INFO
2019-09-24 14:54:55,546 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:54:55,948 [main] INFO
2019-09-24 14:54:55,948 [main] INFO
aliases D1,groupbyid,maxid
2019-09-24 14:54:55,948 [main] INFO
locations: M: D1[1,5],D1[-1,-1],maxid[4,8],groupbyid[3,12] C: maxid[4,8],groupbyid[3,12] R:
maxid[4,8]
2019-09-24 14:54:55,952 [main] INFO
complete
2019-09-24 14:54:55,953 [main] INFO
2019-09-24 14:54:55,961 [Thread-96] INFO
version is 1
OutputCommitter is
map tasks
attempt_local234157501_0004_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

code.
Aliases being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-
1],maxid[4,8],groupbyid[3,12] C: maxid[4,8],groupbyid[3,12] R: maxid[4,8]
26214384(104857536); length = 13/6553600
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine - Aliases
being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-
attempt_local234157501_0004_m_000000_0
executor complete.
reduce tasks
2019-09-24 14:54:56,381 [pool-13-thread-1] INFO
version is 1
ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5f425b3b
2019-09-24 14:54:56,383 [pool-13-thread-1] INFO
attempt_local234157501_0004_m_000000_0
copied.
2019-09-24 14:54:56,392 [pool-13-thread-1] INFO
sorted segments
2019-09-24 14:54:56,394 [pool-13-thread-1] INFO
2019-09-24 14:54:56,394 [pool-13-thread-1] INFO
2019-09-24 14:54:56,394 [pool-13-thread-1] INFO
memory into reduce
sorted segments
copied.
2019-09-24 14:54:56,416 [pool-13-thread-1] INFO
version is 1
2019-09-24 14:54:56,455 [main] INFO
complete
2019-09-24 14:54:56,455 [main] INFO
2019-09-24 14:54:56,475 [pool-13-thread-1] INFO
being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-
copied.
2019-09-24 14:54:56,490 [pool-13-thread-1] INFO
'attempt_local234157501_0004_r_000000_0' to
file:/tmp/temp1989380716/tmp770682788/_temporary/0/task_local234157501_0004_r_000000
reduce > reduce
2019-09-24 14:54:56,498 [Thread-96] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce task
executor complete.
2019-09-24 14:54:57,694 [main] INFO
complete
Script Statistics:

2.7.1 0.16.0 rr 2019-09-24 14:54:55 2019-09-24 14:54:57 GROUP_BY
Success!

Feature Outputs
D1,groupbyid,maxid GROUP_BY,COMBINER file:/tmp/temp1989380716/tmp770682788,
Input(s):
Output(s):
Counters:
Job DAG:
job_local234157501_0004

2019-09-24 14:54:57,962 [main] INFO
2019-09-24 14:54:58,016 [main] INFO
(12,1)
(14,1)
(16,1)
(18,1)
grunt> groupbyname = GROUP D1 BY name;
grunt> dump groupbyname;
2019-09-24 14:56:08,007 [main] INFO
2019-09-24 14:56:08,010 [main] INFO
2019-09-24 14:56:08,017 [main] INFO
2019-09-24 14:56:08,018 [main] INFO
2019-09-24 14:56:08,385 [main] INFO
2019-09-24 14:56:08,385 [main] INFO
2019-09-24 14:56:08,385 [main] INFO
reducer estimator:
2019-09-24 14:56:08,392 [main] INFO
2019-09-24 14:56:08,394 [main] INFO
Parallelism to 1
2019-09-24 14:56:08,404 [main] INFO
single store job
directory: /tmp/1569365768405-0
2019-09-24 14:56:08,536 [main] INFO
PigTextInputFormat
2019-09-24 14:56:08,840 [JobControl] INFO
2019-09-24 14:56:08,900 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:56:09,741 [main] INFO
2019-09-24 14:56:09,741 [main] INFO
aliases D1,groupbyname
2019-09-24 14:56:09,742 [main] INFO
locations: M: D1[1,5],D1[-1,-1],groupbyname[5,14] C: R:
2019-09-24 14:56:09,746 [main] INFO
complete
2019-09-24 14:56:09,746 [main] INFO
2019-09-24 14:56:09,755 [Thread-124] INFO
version is 1
OutputCommitter is
2019-09-24 14:56:09,774 [Thread-124] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting
for map tasks
attempt_local1713093984_0005_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

code.
Aliases being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-
1],groupbyname[5,14] C: R:
26214384(104857536); length = 13/6553600
attempt_local1713093984_0005_m_000000_0
executor complete.
for reduce tasks
2019-09-24 14:56:10,433 [pool-16-thread-1] INFO
version is 1
ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@51b87349
2019-09-24 14:56:10,441 [pool-16-thread-1] INFO
org.apache.hadoop.mapreduce.task.reduce.EventFetcher -
attempt_local1713093984_0005_r_000000_0 Thread started: EventFetcher for fetching Map
Completion Events
attempt_local1713093984_0005_m_000000_0
copied.
2019-09-24 14:56:10,447 [pool-16-thread-1] INFO
sorted segments
2019-09-24 14:56:10,461 [pool-16-thread-1] INFO
2019-09-24 14:56:10,461 [pool-16-thread-1] INFO
2019-09-24 14:56:10,461 [pool-16-thread-1] INFO
memory into reduce
sorted segments
copied.
2019-09-24 14:56:10,463 [pool-16-thread-1] INFO
version is 1
2019-09-24 14:56:10,495 [pool-16-thread-1] INFO
being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1],groupbyname[5,14] C:
R:
copied.
2019-09-24 14:56:10,503 [pool-16-thread-1] INFO
'attempt_local1713093984_0005_r_000000_0' to
file:/tmp/temp1989380716/tmp261580045/_temporary/0/task_local1713093984_0005_r_000000
reduce > reduce
2019-09-24 14:56:10,521 [Thread-124] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce
task executor complete.
2019-09-24 14:56:10,787 [main] INFO
complete
Script Statistics:

2.7.1 0.16.0 rr 2019-09-24 14:56:08 2019-09-24 14:56:10 GROUP_BY
Success!

Feature Outputs
D1,groupbyname GROUP_BY file:/tmp/temp1989380716/tmp261580045,
Input(s):
Output(s):
Counters:
Job DAG:
job_local1713093984_0005

2019-09-24 14:56:10,809 [main] INFO
2019-09-24 14:56:10,846 [main] INFO
( Nagul,{(18, Nagul)})
( Nandhu,{(14, Nandhu)})
( Narien,{(12, Narien)})
( Naveen,{(16, Naveen)})
grunt> groupp = GROUP D1 ALL;
grunt> dump groupp;
2019-09-24 14:57:23,264 [main] INFO
2019-09-24 14:57:23,269 [main] INFO
2019-09-24 14:57:23,275 [main] INFO
2019-09-24 14:57:23,287 [main] INFO
2019-09-24 14:57:23,317 [main] INFO
2019-09-24 14:57:23,318 [main] INFO
2019-09-24 14:57:23,318 [main] INFO
Parallelism to 1
2019-09-24 14:57:23,361 [main] INFO
single store job
directory: /tmp/1569365843362-0
2019-09-24 14:57:23,395 [main] INFO
PigTextInputFormat
2019-09-24 14:57:23,689 [JobControl] INFO
2019-09-24 14:57:23,692 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:57:25,852 [main] INFO
2019-09-24 14:57:25,852 [main] INFO
aliases D1,groupp
2019-09-24 14:57:25,852 [main] INFO
locations: M: D1[1,5],D1[-1,-1],groupp[6,9] C: R:
2019-09-24 14:57:25,859 [main] INFO
complete
2019-09-24 14:57:25,860 [main] INFO
2019-09-24 14:57:25,940 [Thread-152] INFO
version is 1
OutputCommitter is
for map tasks
attempt_local961983538_0006_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

code.
Aliases being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1],groupp[6,9] C:
R:
26214384(104857536); length = 13/6553600
attempt_local961983538_0006_m_000000_0
executor complete.
for reduce tasks
2019-09-24 14:57:26,801 [pool-19-thread-1] INFO
version is 1
ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5ffc30a1
2019-09-24 14:57:26,807 [pool-19-thread-1] INFO
attempt_local961983538_0006_m_000000_0
copied.
2019-09-24 14:57:26,827 [pool-19-thread-1] INFO
sorted segments
2019-09-24 14:57:26,830 [pool-19-thread-1] INFO
2019-09-24 14:57:26,834 [pool-19-thread-1] INFO
2019-09-24 14:57:26,834 [pool-19-thread-1] INFO
memory into reduce
sorted segments
copied.
2019-09-24 14:57:26,836 [pool-19-thread-1] INFO
version is 1
2019-09-24 14:57:26,856 [pool-19-thread-1] INFO
being processed per job phase (AliasName[line,offset]): M: D1[1,5],D1[-1,-1],groupp[6,9] C: R:
copied.
2019-09-24 14:57:26,871 [pool-19-thread-1] INFO
reduce > reduce
2019-09-24 14:57:27,032 [main] INFO
complete
Script Statistics:

2.7.1 0.16.0 rr 2019-09-24 14:57:23 2019-09-24 14:57:27 GROUP_BY
Success!

Feature Outputs
D1,groupp GROUP_BY file:/tmp/temp1989380716/tmp-586598137,
Input(s):
Output(s):
Counters:
Job DAG:
job_local961983538_0006

2019-09-24 14:57:27,065 [main] INFO
2019-09-24 14:57:27,095 [main] INFO
(all,{(18, Nagul),(16, Naveen),(14, Nandhu),(12, Narien)})
grunt> sortbyname = ORDER D1 by id;
grunt> dump sortbyname;
the script: ORDER_BY
2019-09-24 14:58:35,627 [main] INFO
2019-09-24 14:58:35,630 [main] INFO
2019-09-24 14:58:35,692 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizerMR -
Using Secondary Key Optimization for MapReduce node scope-120
2019-09-24 14:58:35,694 [main] INFO
2019-09-24 14:58:35,695 [main] INFO
2019-09-24 14:58:35,783 [main] INFO
2019-09-24 14:58:35,786 [main] INFO
single store job
directory: /tmp/1569365915800-0
2019-09-24 14:58:35,837 [main] INFO
PigTextInputFormat
2019-09-24 14:58:36,000 [JobControl] INFO
2019-09-24 14:58:36,003 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:58:36,306 [Thread-180] INFO
version is 1
OutputCommitter is
for map tasks
attempt_local2027325755_0007_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

version is 1
2019-09-24 14:58:36,366 [main] INFO
2019-09-24 14:58:36,366 [main] INFO
aliases D1
2019-09-24 14:58:36,367 [main] INFO
locations: M: D1[1,5],D1[-1,-1] C: R:
2019-09-24 14:58:36,368 [main] INFO
complete
2019-09-24 14:58:36,368 [main] INFO
code.
commit now
file:/tmp/temp1989380716/tmp2118310213/_temporary/0/task_local2027325755_0007_m_000000
attempt_local2027325755_0007_m_000000_0
executor complete.
2019-09-24 14:58:36,607 [main] INFO
complete
2019-09-24 14:58:36,624 [main] INFO
2019-09-24 14:58:36,630 [main] INFO
2019-09-24 14:58:36,638 [main] INFO
reducer estimator:
2019-09-24 14:58:36,648 [main] INFO
2019-09-24 14:58:36,649 [main] INFO
Parallelism to 1
2019-09-24 14:58:36,650 [main] INFO
single store job
2019-09-24 14:58:36,705 [main] INFO
2019-09-24 14:58:36,808 [JobControl] INFO
2019-09-24 14:58:36,808 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:58:37,170 [Thread-200] INFO
version is 1
OutputCommitter is
for map tasks
attempt_local2104222054_0008_m_000000_0
version is 1
Total Length = 63
Input split[0]:
Length = 63
Locations:
-----------------------

being processed file:/tmp/temp1989380716/tmp2118310213/part-m-00000:0+63
2019-09-24 14:58:37,393 [main] INFO
2019-09-24 14:58:37,399 [main] INFO
aliases sortbyname
2019-09-24 14:58:37,401 [main] INFO
locations: M: sortbyname[7,13] C: R:
2019-09-24 14:58:37,609 [LocalJobRunner Map Task Executor #0] WARN
org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
Aliases being processed per job phase (AliasName[line,offset]): M: sortbyname[7,13] C: R:
26214384(104857536); length = 13/6553600
attempt_local2104222054_0008_m_000000_0
executor complete.
for reduce tasks
2019-09-24 14:58:37,637 [pool-24-thread-1] INFO
version is 1
ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7665dca7
2019-09-24 14:58:37,652 [pool-24-thread-1] INFO
Completion Events
attempt_local2104222054_0008_m_000000_0
copied.
2019-09-24 14:58:37,668 [pool-24-thread-1] INFO
sorted segments
2019-09-24 14:58:37,676 [pool-24-thread-1] INFO
2019-09-24 14:58:37,677 [pool-24-thread-1] INFO
2019-09-24 14:58:37,677 [pool-24-thread-1] INFO
memory into reduce
sorted segments
copied.
2019-09-24 14:58:37,683 [pool-24-thread-1] INFO
version is 1
2019-09-24 14:58:37,720 [pool-24-thread-1] INFO
being processed per job phase (AliasName[line,offset]): M: sortbyname[7,13] C: R:
copied.
2019-09-24 14:58:37,735 [pool-24-thread-1] INFO
reduce > reduce
2019-09-24 14:58:37,970 [main] INFO
complete
2019-09-24 14:58:38,011 [main] INFO
2019-09-24 14:58:38,011 [main] INFO
2019-09-24 14:58:38,014 [main] INFO
Parallelism to 1
2019-09-24 14:58:38,019 [main] INFO
single store job
2019-09-24 14:58:38,049 [main] INFO
2019-09-24 14:58:38,118 [JobControl] INFO
2019-09-24 14:58:38,118 [JobControl] INFO
process : 1
splits:1
2019-09-24 14:58:38,464 [Thread-226] INFO
version is 1
OutputCommitter is
for map tasks
attempt_local2057231340_0009_m_000000_0
version is 1
Total Length = 63
Input split[0]:
Length = 63
Locations:
-----------------------

being processed file:/tmp/temp1989380716/tmp2118310213/part-m-00000:0+63
2019-09-24 14:58:38,682 [main] INFO
2019-09-24 14:58:38,682 [main] INFO
aliases sortbyname
2019-09-24 14:58:38,682 [main] INFO
locations: M: sortbyname[7,13] C: R:
2019-09-24 14:58:38,835 [LocalJobRunner Map Task Executor #0] WARN
org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
Aliases being processed per job phase (AliasName[line,offset]): M: sortbyname[7,13] C: R:
26214384(104857536); length = 13/6553600
attempt_local2057231340_0009_m_000000_0
executor complete.
for reduce tasks
2019-09-24 14:58:38,882 [pool-27-thread-1] INFO
version is 1
ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@8198439
2019-09-24 14:58:38,888 [pool-27-thread-1] INFO
Completion Events
attempt_local2057231340_0009_m_000000_0
copied.
2019-09-24 14:58:38,899 [pool-27-thread-1] INFO
sorted segments
2019-09-24 14:58:38,906 [pool-27-thread-1] INFO
2019-09-24 14:58:38,907 [pool-27-thread-1] INFO
2019-09-24 14:58:38,907 [pool-27-thread-1] INFO
memory into reduce
sorted segments
copied.
2019-09-24 14:58:38,911 [pool-27-thread-1] INFO
version is 1
2019-09-24 14:58:38,990 [pool-27-thread-1] INFO
being processed per job phase (AliasName[line,offset]): M: sortbyname[7,13] C: R:
copied.
2019-09-24 14:58:38,996 [pool-27-thread-1] INFO
reduce > reduce
2019-09-24 14:58:39,150 [main] INFO
complete
Script Statistics:
2.7.1 0.16.0 rr 2019-09-24 14:58:35 2019-09-24 14:58:39 ORDER_BY
Success!

Feature Outputs
D1MAP_ONLY
sortbyname ORDER_BY file:/tmp/temp1989380716/tmp-1327869809,
sortbyname SAMPLER
Input(s):
Output(s):
Counters:
Job DAG:
job_local2027325755_0007 -> job_local2104222054_0008,
job_local2104222054_0008 -> job_local2057231340_0009,
job_local2057231340_0009

2019-09-24 14:58:39,227 [main] INFO
2019-09-24 14:58:39,273 [main] INFO
(12, Narien)
(14, Nandhu)
(16, Naveen)
(18, Nagul)
grunt> explain D1;
2019-09-24 14:59:09,956 [main] INFO
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
D1: (Name: LOStore Schema: id#441:int,name#442:chararray)
|
|---D1: (Name: LOForEach Schema: id#441:int,name#442:chararray)
| |
| (Name: LOGenerate[false,false] Schema:
id#441:int,name#442:chararray)ColumnPrune:InputUids=[441, 442]ColumnPrune:OutputUids=[441,
442]
| | |
| | (Name: Cast Type: int Uid: 441)
| | |
| | |---id:(Name: Project Type: bytearray Uid: 441 Input: 0 Column: (*))
| | |
| | (Name: Cast Type: chararray Uid: 442)
| | |
| | |---name:(Name: Project Type: bytearray Uid: 442 Input: 1 Column: (*))
| |
| |---(Name: LOInnerLoad[0] Schema: id#441:bytearray)
| |
| |---(Name: LOInnerLoad[1] Schema: name#442:bytearray)
|
|---D1: (Name: LOLoad Schema: id#441:bytearray,name#442:bytearray)RequiredFields:null
#-----------------------------------------------
# Physical Plan:
#-----------------------------------------------
D1: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-149
|
|---D1: New For Each(false,false)[bag] - scope-148
| |
| Cast[int] - scope-143
| |
| |---Project[bytearray][0] - scope-142
| |
| Cast[chararray] - scope-146
| |
|
|---D1: Load(hdfs://localhost:54310/dir11/emp.txt:PigStorage(',')) - scope-141
2019-09-24 14:59:09,960 [main] INFO

2019-09-24 14:59:09,961 [main] INFO
2019-09-24 14:59:09,961 [main] INFO
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node scope-150
Map Plan
D1: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-149
|
|---D1: New For Each(false,false)[bag] - scope-148
| |
| Cast[int] - scope-143
| |
| |
| Cast[chararray] - scope-146
| |
|
|---D1: Load(hdfs://localhost:54310/dir11/emp.txt:PigStorage(',')) - scope-141--------
Global sort: false
----------------
grunt> Illustrate D1;

2019-09-24 15:00:02,029 [main] INFO
system at: file:///
2019-09-24 15:00:02,327 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[ConstantCalculator, LoadTypeCastInserter, PredicatePushdownOptimizer,
StreamTypeCastInserter], RULES_DISABLED=[AddForEach, ColumnMapKeyPrune,
GroupByConstParallelSetter, LimitOptimizer, MergeFilter, MergeForEach, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter]}
2019-09-24 15:00:02,381 [main] INFO
2019-09-24 15:00:02,387 [main] INFO
2019-09-24 15:00:02,387 [main] INFO
2019-09-24 15:00:02,388 [main] INFO
2019-09-24 15:00:02,501 [main] INFO
processed per job phase (AliasName[line,offset]): M: D1[1,5] C: R:
2019-09-24 15:00:02,503 [main] INFO org.apache.pig.builtin.PigStorage - Using PigTextInputFormat
2019-09-24 15:00:02,506 [main] INFO
2019-09-24 15:00:03,124 [main] INFO
2019-09-24 15:00:03,126 [main] INFO
2019-09-24 15:00:03,126 [main] INFO
2019-09-24 15:00:03,128 [main] INFO
2019-09-24 15:00:03,159 [main] INFO
2019-09-24 15:00:03,184 [main] INFO
2019-09-24 15:00:03,185 [main] INFO
2019-09-24 15:00:03,186 [main] INFO
2019-09-24 15:00:03,189 [main] INFO
2019-09-24 15:00:03,213 [main] INFO
2019-09-24 15:00:03,214 [main] INFO
2019-09-24 15:00:03,215 [main] INFO
2019-09-24 15:00:03,216 [main] INFO
2019-09-24 15:00:03,217 [main] INFO
2019-09-24 15:00:03,254 [main] INFO
2019-09-24 15:00:03,259 [main] INFO
2019-09-24 15:00:03,260 [main] INFO
2019-09-24 15:00:03,260 [main] INFO
2019-09-24 15:00:03,261 [main] INFO
2019-09-24 15:00:03,865 [main] INFO
(16, Naveen)
2019-09-24 15:00:03,950 [main] INFO
2019-09-24 15:00:03,950 [main] INFO
2019-09-24 15:00:03,951 [main] INFO
2019-09-24 15:00:03,951 [main] INFO
2019-09-24 15:00:04,633 [main] INFO
2019-09-24 15:00:04,640 [main] INFO
2019-09-24 15:00:04,641 [main] INFO
2019-09-24 15:00:04,641 [main] INFO
2019-09-24 15:00:04,647 [main] INFO
2019-09-24 15:00:05,293 [main] INFO
2019-09-24 15:00:05,295 [main] INFO
2019-09-24 15:00:05,298 [main] INFO
2019-09-24 15:00:05,298 [main] INFO
2019-09-24 15:00:05,319 [main] INFO
2019-09-24 15:00:08,770 [main] INFO
2019-09-24 15:00:08,771 [main] INFO
2019-09-24 15:00:08,772 [main] INFO
2019-09-24 15:00:08,772 [main] INFO
2019-09-24 15:00:08,774 [main] INFO
2019-09-24 15:00:11,947 [main] INFO
--------------------------------------------
| D1 | id:int | name:chararray |
--------------------------------------------
| | 16 | Naveen |
--------------------------------------------
grunt> STORE D1 INTO 'hdfs://localhost:54310/dir11/stu';

mapred.textoutputformat.separator is deprecated. Instead, use
mapreduce.output.textoutputformat.separator
the script: UNKNOWN
2019-09-24 15:01:40,817 [main] INFO
2019-09-24 15:01:40,820 [main] INFO
2019-09-24 15:01:40,822 [main] INFO
2019-09-24 15:01:40,826 [main] INFO
2019-09-24 15:01:40,855 [main] INFO
2019-09-24 15:01:40,856 [main] INFO
single store job
directory: /tmp/1569366100857-0
2019-09-24 15:01:40,867 [main] INFO
PigTextInputFormat
2019-09-24 15:01:41,107 [JobControl] INFO
2019-09-24 15:01:41,110 [JobControl] INFO
process : 1
splits:1
2019-09-24 15:01:41,945 [main] INFO
2019-09-24 15:01:41,945 [main] INFO
aliases D1
2019-09-24 15:01:41,946 [main] INFO
locations: M: D1[1,5],D1[-1,-1] C: R:
2019-09-24 15:01:41,950 [main] INFO
complete
2019-09-24 15:01:41,950 [main] INFO
mapred.textoutputformat.separator is deprecated. Instead, use
mapreduce.output.textoutputformat.separator
2019-09-24 15:01:42,001 [Thread-257] INFO
version is 1
OutputCommitter is
for map tasks
attempt_local1377189287_0010_m_000000_0
version is 1
Total Length = 43
Input split[0]:
Length = 43
Locations:
-----------------------

version is 1
code.
commit now
hdfs://localhost:54310/dir11/stu/_temporary/0/task_local1377189287_0010_m_000000
attempt_local1377189287_0010_m_000000_0
executor complete.
2019-09-24 15:01:44,503 [main] INFO
complete
Script Statistics:

2.7.1 0.16.0 rr 2019-09-24 15:01:40 2019-09-24 15:01:44 UNKNOWN
Success!

Feature Outputs
D1MAP_ONLY hdfs://localhost:54310/dir11/stu,
Input(s):
Output(s):
Successfully stored 4 records (43 bytes) in: "hdfs://localhost:54310/dir11/stu"
Counters:
Job DAG:
job_local1377189287_0010

2019-09-24 15:01:44,545 [main] INFO
grunt>

Pig Program - Odt

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pig Program - Odt

Uploaded by

Copyright:

Available Formats

rr@ubuntu:~$ start-all.

Details at logfile: /home/rr/pig_1569364530821.log

Details at logfile: /home/rr/pig_1569364530821.log

2019-09-24 14:49:09,419 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 14:49:10,800 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:50:28,567 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 14:50:28,907 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:52:56,780 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 14:53:03,359 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:54:56,006 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 14:54:57,905 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:56:09,802 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 14:56:10,795 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:57:26,131 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 14:57:27,038 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:58:36,351 [LocalJobRunner Map Task Executor #0] INFO

2019-09-24 14:58:37,206 [LocalJobRunner Map Task Executor #0] INFO

2019-09-24 14:58:38,515 [LocalJobRunner Map Task Executor #0] INFO

Job Stats (time in seconds):

2019-09-24 14:58:39,159 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

2019-09-24 14:59:09,960 [main] INFO

grunt> Illustrate D1;

grunt> STORE D1 INTO 'hdfs://localhost:54310/dir11/stu';

2019-09-24 15:01:42,570 [LocalJobRunner Map Task Executor #0] INFO

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

Job Stats (time in seconds):

2019-09-24 15:01:44,507 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize

You might also like