Professional Documents
Culture Documents
Hive Icde2010 PDF
Hive Icde2010 PDF
Hadoop
Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Antony, Hao Liu
and Raghotham Murthy
Facebook Data Infrastructure Team
Abstract The size of data sets being collected and analyzed in
the industry for business intelligence is growing rapidly, making
traditional warehousing solutions prohibitively expensive.
Hadoop [1] is a popular open-source map-reduce implementation
which is being used in companies like Yahoo, Facebook etc. to
store and process extremely large data sets on commodity
hardware. However, the map-reduce programming model is very
low level and requires developers to write custom programs
which are hard to maintain and reuse. In this paper, we present
Hive, an open-source data warehousing solution built on top of
Hadoop. Hive supports queries expressed in a SQL-like
declarative language - HiveQL, which are compiled into mapreduce jobs that are executed using Hadoop. In addition, HiveQL
enables users to plug in custom map-reduce scripts into queries.
The language includes a type system with support for tables
containing primitive types, collections like arrays and maps, and
nested compositions of the same. The underlying IO libraries can
be extended to query data in custom formats. Hive also includes
a system catalog - Metastore that contains schemas and
statistics, which are useful in data exploration, query
optimization and query compilation. In Facebook, the Hive
warehouse contains tens of thousands of tables and stores over
700TB of data and is being used extensively for both reporting
and ad-hoc analyses by more than 200 users per month.
I. INTRODUCTION
Scalable analysis on large data sets has been core to the
functions of a number of teams at Facebook - both
engineering and non-engineering. Apart from ad hoc analysis
and business intelligence applications used by analysts across
the company, a number of Facebook products are also based
on analytics. These products range from simple reporting
applications like Insights for the Facebook Ad Network, to
more advanced kind such as Facebook's Lexicon product [2].
As a result a flexible infrastructure that caters to the needs of
these diverse applications and users and that also scales up in
a cost effective manner with the ever increasing amounts of
data being generated on Facebook, is critical. Hive and
Hadoop are the technologies that we have used to address
these requirements at Facebook.
The entire data processing infrastructure in Facebook prior
to 2008 was built around a data warehouse built using a
commercial RDBMS. The data that we were generating was
growing very fast - as an example we grew from a 15TB data
set in 2007 to a 700TB data set today. The infrastructure at
that time was so inadequate that some daily data processing
jobs were taking more than a day to process and the situation
was just getting worse with every passing day. We had an
urgent need for infrastructure that could scale along with our
The final storage unit concept that Hive uses is the concept
of Buckets. A bucket is a file within the leaf level directory of
a table or a partition. At the time the table is created, the user
can specify the number of buckets needed and the column on
which to bucket the data. In the current implementation this
information is used to prune the data in case the user runs the
query on a sample of data e.g. a table that is bucketed into 32
buckets can quickly generate a 1/32 sample by choosing to
look at the first bucket of data. Similarly, the statement
SELECT * FROM t TABLESAMPLE(2 OUT OF 32);
would scan the data present in the second bucket. Note that
the onus of ensuring that the bucket files are properly created
and named are a responsibility of the application and HiveQL
DDL statements do not currently try to bucket the data in a
way that it becomes compatible to the table properties.
Consequently, the bucketing information should be used with
caution.
Though the data corresponding to a table always resides in
the <warehouse_root_directory>/test_table location in hdfs,
Hive also enables users to query data stored in other locations
in hdfs. This can be achieved through the EXTERNAL
TABLE clause as shown in the following example.
CREATE EXTERNAL TABLE test_extern(c1 string, c2 int)
LOCATION '/user/mytables/mydata';
With this statement, the user is able to specify that test_extern
is an external table with each row comprising of two columns
c1 and c2. In addition the data files are stored in the location
/user/mytables/mydata in hdfs. Note that as no custom SerDe
has been defined it is assumed that the data is in Hives
internal format. An external table differs from a normal table
in only that a drop table command on an external table only
drops the table metadata and does not delete any data. A drop
on a normal table on the other hand drops the data associated
with the table as well.
B. Serialization/Deserialization (SerDe)
As mentioned previously Hive can take an implementation
of the SerDe java interface provided by the user and associate
it to a table or partition. As a result custom data formats can
easily be interpreted and queried from. The default SerDe
implementation in Hive is called the LazySerDe it
deserializes rows into internal objects lazily so that the cost of
deserialization of a column is incurred only if the column of
the row is needed in some query expression. The LazySerDe
assumes that the data is stored in the file such that the rows
are delimited by a newline (ascii code 13) and the columns
within a row are delimited by ctrl-A (ascii code 1). This SerDe
can also be used to read data that uses any other delimiter
character between columns.
set hive.groupby.skewindata=true;
SELECT t1.c1, sum(t1.c2)
FROM t1
GROUP BY t1;
ii. Hash based partial aggregations in the
mappers Hash based partial aggregations
can potentially reduce the data that is sent
by the mappers to the reducers. This in turn
reduces the amount of time spent in sorting
and merging this data. As a result a lot of
performance gains can be achieved using
this strategy. Hive enables users to control
the amount of memory that can be used on
the mapper to hold the rows in a hash table
for this optimization. The parameter
hive.map.aggr.hash.percentmemory
specifies the fraction of mapper memory
that can be used to hold the hash table, e.g.
0.5 would ensure that as soon as the hash
table size exceeds half of the maximum
memory for a mapper, the partial aggregates
stored therein are sent to the reducers. The
parameter hive.map.aggr.hash.min.reduction
is also used to control the amount of
memory used in the mappers.
make sure that the join is performed only once. The plan for
the query is shown in Fig 3 below.
The nodes in the plan are physical operators and the edges
represent the flow of data between operators. The last line in
each node represents the output schema of that operator. For
lack of space, we do not describe the parameters specified
within each operator node. The plan has three map-reduce
jobs.
Fig. 3: Query plan for multi-table insert query with 3 map/reduce jobs
C. Execution Engine
Finally the tasks are executed in the order of their
dependencies. Each dependent task is only executed if all of
its prerequisites have been executed. A map/reduce task first
serializes its part of the plan into a plan.xml file. This file is
then added to the job cache for the task and instances of
ExecMapper and ExecReducers are spawned using Hadoop.
Each of these classes deserializes the plan.xml and executes
the relevant part of the operator DAG. The final results are
stored in a temporary location. At the end of the entire query,
the final data is moved to the desired location in case of
DMLs. In the case of queries the data is served as such from
the temporary location.
VII.
CONCLUSIONS AND FUTURE WORK
Hive is a work in progress. It is an open-source project, and
is being actively worked on by Facebook as well as several
external contributors.
HiveQL currently accepts only a subset of SQL as valid
queries. We are working towards making HiveQL subsume
SQL syntax. Hive currently has a naive rule-based optimizer
with a small number of simple rules. We plan to build a costbased optimizer and adaptive optimization techniques to come
up with more efficient plans. We are exploring columnar
storage and more intelligent data placement to improve scan
performance. We are running performance benchmarks based
on [9] to measure our progress as well as compare against
other systems. In our preliminary experiments, we have been
able to improve the performance of Hadoop itself by ~20%
compared to [9]. The improvements involved using faster
Hadoop data structures to process the data, for example, using
Text instead of String. The same queries expressed easily in
HiveQL had ~20% overhead compared to our optimized
Hadoop implementation, i.e., Hive's performance is on par
with the Hadoop code from [9]. We have also run the industry
standard decision support benchmark TPC-H [11]. Based on
these experiments, we have identified several areas for
performance improvement and have begun working on them.
More details are available in [10] and [12]. We are enhancing
the JDBC and ODBC drivers for Hive for integration with
commercial BI tools that only work with traditional relational
warehouses. We are exploring methods for multi-query
optimization techniques and performing generic n-way joins
in a single map-reduce job.
ACKNOWLEDGMENT
We would like to thank our user and developer community for
their contributions, with special thanks to Eric Hwang, Yuntao
Jia, Yongqiang He, Edward Capriolo, and Dhruba Borthakur.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]