ClouderaCDH搭建_cloudera镜像资源-CSDN文库

hadoop

4星 · 超过85%的资源需积分: 10 46 浏览量 2013-08-10 06:55:00 上传评论 1 收藏 2.92MB PDF 举报

资源推荐

资源详情

资源评论

CDH4 Installation Guide

Table of Contents

About this Guide ......................................................................................................15

What's New in CDH4................................................................................................17

Before You Install CDH4 on a Cluster ...................................................................19

CDH4 Installation.....................................................................................................21

CDH4 and MapReduce ..............................................................................................................................21

MapReduce 2.0 (YARN) .........................................................................................................................................21

Ways To Install CDH4 ...............................................................................................................................21

How Packaging Affects CDH4 Deployment ........................................................................................................22

Before You Begin Installing CDH4 Manually...........................................................................................22

Installing CDH4..........................................................................................................................................23

Step 1: Add or Build the CDH4 Repository or Download the "1-click Install" package. ..................................23

Step 1a: Optionally Add a Repository Key............................................................................................................26

Step 2: Install CDH4 with MRv1.............................................................................................................................26

Step 3: Install CDH4 with YARN.............................................................................................................................28

Step 4: (Optional) Install LZO.................................................................................................................................30

Step 5: Deploy CDH and Install Components.......................................................................................................31

Installing CDH4 Components...................................................................................................................31

Viewing the Apache Hadoop Documentation.........................................................................................32

Installing an Earlier CDH4 Release........................................................................33

Downloading and Installing an Earlier Release......................................................................................33

On Red Hat-compatible systems.........................................................................................................................33

On SLES systems....................................................................................................................................................34

On Ubuntu and Debian systems...........................................................................................................................35

Upgrading from CDH3 to CDH4...............................................................................37

CDH4 and MapReduce...............................................................................................................................37

MapReduce 2.0 (YARN)..........................................................................................................................................37

High Availability......................................................................................................................................................38

Before You Begin.......................................................................................................................................38

Plan Downtime.......................................................................................................................................................38

Considerations for Secure Clusters......................................................................................................................38

Upgrading to CDH4....................................................................................................................................39

Step 1: Back Up Configuration Data and Uninstall Components.......................................................................39

Step 2: Back up the HDFS Metadata.....................................................................................................................40

Step 3: Copy the Hadoop Configuration to the Correct Location and Update Alternatives............................40

Step 4: Uninstall CDH3 Hadoop.............................................................................................................................41

Step 5: Download CDH4..........................................................................................................................................42

Step 6a: Install CDH4 with MRv1..........................................................................................................................44

Step 6b: Install CDH4 with YARN..........................................................................................................................45

Step 7: Copy the CDH4 Logging File......................................................................................................................46

Step 7a: (Secure Clusters Only) Set Variables for Secure DataNodes...............................................................47

Step 8: Upgrade the HDFS Metadata....................................................................................................................47

Step 9: Create the HDFS /tmp Directory..............................................................................................................48

Step 10: Start MapReduce (MRv1) or YARN.........................................................................................................48

Step 11: Set the Sticky Bit......................................................................................................................................52

Step 12: Re-Install CDH4 Components.................................................................................................................52

Step 13: Apply Configuration File Changes..........................................................................................................52

Step 14: Finalize the HDFS Metadata Upgrade...................................................................................................53

Migrating data between a CDH3 and CDH4 cluster.............................................55

Requirements............................................................................................................................................55

Using DistCp to Migrate Data between two Clusters............................................................................55

The DistCp Command.............................................................................................................................................55

Post-migration Verification......................................................................................................................56

Upgrading from an Earlier CDH4 Release ............................................................57

Before You Begin.......................................................................................................................................57

Upgrading to the Latest Version of CDH4...............................................................................................58

Step 1: Prepare the cluster for the upgrade........................................................................................................58

Step 2: Download the CDH4 package on each of the hosts in your cluster.....................................................59

Step 3: Upgrade the packages on the appropriate hosts...................................................................................61

Step 4: Upgrade the HDFS Metadata (Beta 1 or earlier).....................................................................................64

Step 5: Start HDFS (Beta 2 or later).......................................................................................................................65

Step 5a: Verify that /tmp Exists and Has the Right Permissions.....................................................................65

Step 6: Start MapReduce (MRv1) or YARN...........................................................................................................66

Step 7: Set the Sticky Bit........................................................................................................................................70

Step 8: Upgrade Components to CDH4.................................................................................................................71

Step 9: Apply Configuration File Changes............................................................................................................71

Step 10: Finalize the HDFS Metadata Upgrade (Beta 1 or earlier)....................................................................71

Configuring Ports for CDH4.....................................................................................73

Ports Used by Components of CDH4.......................................................................................................73

Ports Used by Third Parties......................................................................................................................77

Deploying CDH4 in Pseudo-Distributed Mode.....................................................79

Deploying CDH4 on a Cluster .................................................................................81

Configuring Network Names....................................................................................................................81

Deploying HDFS on a Cluster ...................................................................................................................82

Copying the Hadoop Configuration ......................................................................................................................82

Customizing Configuration Files ..........................................................................................................................83

Configuring Local Storage Directories .................................................................................................................84

Configuring DataNodes to Tolerate Local Storage Directory Failure ...............................................................86

Formatting the NameNode ..................................................................................................................................86

Configuring a Remote NameNode Storage Directory ........................................................................................87

Configuring the Secondary NameNode ...............................................................................................................87

Enabling Trash .......................................................................................................................................................89

Configuring Storage-Balancing for the DataNodes............................................................................................90

Enabling WebHDFS ................................................................................................................................................90

Configuring LZO .....................................................................................................................................................90

Deploy MRv1 or YARN ...........................................................................................................................................91

Deploying MapReduce v1 (MRv1) on a Cluster.......................................................................................91

Step 1: Configuring Properties for MRv1 Clusters..............................................................................................92

Step 2: Configure Local Storage Directories for Use by MRv1 Daemons..........................................................92

Step 3: Configure a Health Check Script for DataNode Processes....................................................................93

Step 4: Configure JobTracker Recovery.................................................................................................................94

Enabling JobTracker Recovery...............................................................................................................................94

Step 5: Deploy your Custom Configuration to your Entire Cluster....................................................................94

Step 6: Start HDFS on Every Node in the Cluster................................................................................................95

Step 7: Create the HDFS /tmp Directory..............................................................................................................95

Step 8: Create MapReduce /var directories.........................................................................................................95

Step 9: Verify the HDFS File Structure..................................................................................................................95

Step 10: Create and Configure the mapred.system.dir Directory in HDFS.......................................................95

Step 11: Start MapReduce.....................................................................................................................................96

Step 12: Create a Home Directory for each MapReduce User............................................................................96

Configure the Hadoop Daemons to Start at Boot Time......................................................................................96

Deploying MapReduce v2 (YARN) on a Cluster.......................................................................................96

About MapReduce v2 (YARN)................................................................................................................................97

Step 1: Configure Properties for YARN Clusters..................................................................................................97

Step 2: Configure YARN daemons.........................................................................................................................98

Step 3: Configure the History Server..................................................................................................................101

Step 4: Configure the Staging Directory.............................................................................................................101

Step 5: Deploy your custom Configuration to your Entire Cluster..................................................................102

Step 6: Start HDFS on Every Node in the Cluster..............................................................................................102

Step 7: Create the HDFS /tmp Directory............................................................................................................102