hadoop mapred_tutorial官方文档

Hadoop Map-Reduce Tutorial

Table of contents

1 Purpose...............................................................................................................................2

2 Pre-requisites......................................................................................................................2

3 Overview............................................................................................................................2

4 Inputs and Outputs............................................................................................................. 3

5 Example: WordCount v1.0................................................................................................ 3

5.1 Source Code...................................................................................................................3

5.2 Usage.............................................................................................................................6

6.5 Job Input......................................................................................................................16

6.6 Job Output................................................................................................................... 17

6.7 Other Useful Features..................................................................................................18

7 Example: WordCount v2.0.............................................................................................. 22

7.1 Source Code.................................................................................................................22

7.2 Sample Runs................................................................................................................28

7.3 Highlights.................................................................................................................... 30

Copyright © 2007 The Apache Software Foundation. All rights reserved.

1. Purpose

This document comprehensively describes all user-facing facets of the Hadoop Map-Reduce

Hadoop Cluster Setup for large, distributed clusters.

3. Overview

Hadoop Map-Reduce is a software framework for easily writing applications which process

vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of

nodes) of commodity hardware in a reliable, fault-tolerant manner.

A Map-Reduce job usually splits the input data-set into independent chunks which are

processed by the map tasks in a completely parallel manner. The framework sorts the outputs

of the maps, which are then input to the reduce tasks. Typically both the input and the output

of the job are stored in a file-system. The framework takes care of scheduling tasks,

monitoring them and re-executes the failed tasks.

component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves

execute the tasks as directed by the master.

Minimally, applications specify the input/output locations and supply map and reduce

functions via implementations of appropriate interfaces and/or abstract-classes. These, and

other job parameters, comprise the job configuration. The Hadoop job client then submits the

job (jar/executable etc.) and configuration to the JobTracker which then assumes the

responsibility of distributing the software/configuration to the slaves, scheduling tasks and

monitoring them, providing status and diagnostic information to the job-client.

Although the Hadoop framework is implemented in JavaTM, Map-Reduce applications need

Hadoop Map-Reduce Tutorial

•

Hadoop Pipes is a SWIG- compatible C++ API to implement Map-Reduce applications

(non JNITM based).

4. Inputs and Outputs

The Map-Reduce framework operates exclusively on <key, value> pairs, that is, the

framework views the input to the job as a set of <key, value> pairs and produces a set of

<key, value> pairs as the output of the job, conceivably of different types.

The key and value classes have to be serializable by the framework and hence need to

implement the Writable interface. Additionally, the key classes have to implement the

WritableComparable interface to facilitate sorting by the framework.

Input and Output types of a Map-Reduce job:

given input set.

This works with a local-standalone, pseudo-distributed or fully-distributed Hadoop

installation.

Hadoop Map-Reduce Tutorial

Hadoop Map-Reduce Tutorial

Hadoop Map-Reduce Tutorial