Facebook | Data/Hive/Hive 2.0 Tutorial - Facebook http://www.dev.facebook.com/intern/wiki/index.php/Data/Hive/Hive_2...
1 of 10 7/8/2008 3:40 PM
[edit]
[edit]
[edit]
[edit]
Data/Hive/Hive 2.0 Tutorial
Concepts
What is Hive 2.0
Hive 2.0 is the next generation infrastructure made with the
goal of providing tools to enable easy data summarization,
adhoc querying and analysis of detail data. Just like Hive 1.0,
Hive 2.0 provides a mechanism to put structure on the data. In
addition it also provides a simple query language called QL
which is based on SQL and which enables users familiar with
SQL to do adhoc querying, summarization and data analysis. At
the same time, this language also allows traditional
map/reduce programmers to be able to plug in their custom
mappers and reducers to do more sophisticated analysis which
may not be supported by the built in capabilities of the
language.
What is NOT Hive 2.0
Hive is based on hadoop which is a batch processing system.
Accordingly, this system does not and cannot promise low
latencies on queries. The paradigm here is strictly of submitting
jobs and being notified when the jobs are completed as
opposed to real time queries. As a result it should not be
compared with systems like Oracle where analysis is done on a
significantly smaller amount of data but the analysis proceeds
much more iteratively with the response times between iterations being less than a few minutes. For
Hive queries response times for even the smallest jobs can be of the order of 5-10 minutes and for
larger jobs this may even run into hours.
In the following sections we provide a tutorial on the capabilities of the system. We start by
describing the concepts of data types, tables and partitions (which are very similar to what you would
find in a traditional relational database) and then illustrate the capabilities of the language with the
help of some examples:
Data Units
In order of granularity - Hive data is organized into:
Tables: Homogenous units of data which have the same schema. An example of a table could be
page_views table. Each row of this table could comprise of the following columns, which define
the schema of the table:
timestamp - which corresponds to a datetime type that captures when the page was viewed.
userid - which is a long integer identifying the user who viewed the page.
page_url - which is a string that captures the location of the page.
referer_url - which is a string that captures the location of the page from where we arrived
at the current page.
IP - which is a string that captures the IP address from where the page request was made.
Partitions: Each Table can have one or more partition Keys which determines how the data is
stored. Partitions - apart from being storage units - also allow the user to efficiently identify the
rows that satisfy a certain criteria. e.g a date_ partition of type Datestamp and country_ partition
of type String. Each unique specification of the partition keys defines a partition of the Table e.g.
all "US" data from "2008-02-02" is a partition of the page_views table. Therefore, if you have to
run analysis on only the "US" data for 2008-02-02, you can run that analysis only on the relevant
Concepts
What is Hive 2.0
What is NOT Hive 2.0
Data Units
Type System
Language capabilities
Usage and Examples
Creating Tables
Describing and Showing
Tables
Loading Data
Simple Query
Partition Based Query
Joins
Aggregations
Multi Table/File Inserts
Inserting into local files
Sampling
Union all
Array Operations
Map Operations
Custom map/reduce scripts
Co groups
Useful built in Functions
Altering Tables
Logged in as Ashish Thusoo LogoutTracCamp Tasks Release Trac Wiki Data
Wiki
Article
Edit History Delete Move Watch
Search
