没有合适的资源?快使用搜索试试~ 我知道了~
Cloudera CDH搭建
4星 · 超过85%的资源 需积分: 10 40 下载量 178 浏览量
2013-08-10
06:55:00
上传
评论 1
收藏 2.92MB PDF 举报
温馨提示
试读
294页
cloudera hadoop搭建手册,CDH包括hdfs mapreduce hbase hive oozie sqoop zookeeper pig 等
资源推荐
资源详情
资源评论
CDH4 Installation Guide
Important Notice
(c) 2010-2013 Cloudera, Inc. All rights reserved.
Cloudera, the Cloudera logo, Cloudera Impala, and any other product or service names or slogans
contained in this document are trademarks of Cloudera and its suppliers or licensors, and may
not be copied, imitated or used, in whole or in part, without the prior written permission of
Cloudera or the applicable trademark holder.
Hadoop and the Hadoop elephant logo are trademarks of the Apache Software Foundation. All
other trademarks, registered trademarks, product names and company names or logos
mentioned in this document are the property of their respective owners. Reference to any
products, services, processes or other information, by trade name, trademark, manufacturer,
supplier or otherwise does not constitute or imply endorsement, sponsorship or recommendation
thereof by us.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting
the rights under copyright, no part of this document may be reproduced, stored in or introduced
into a retrieval system, or transmitted in any form or by any means (electronic, mechanical,
photocopying, recording, or otherwise), or for any purpose, without the express written permission
of Cloudera.
Cloudera may have patents, patent applications, trademarks, copyrights, or other intellectual
property rights covering subject matter in this document. Except as expressly provided in any
written license agreement from Cloudera, the furnishing of this document does not give you
any license to these patents, trademarks copyrights, or other intellectual property.
The information in this document is subject to change without notice. Cloudera shall not be
liable for any damages resulting from technical errors or omissions which may be present in
this document, or from use of this document.
Cloudera, Inc.
220 Portage Avenue
Palo Alto, CA 94306
info@cloudera.com
US: 1-888-789-1488
Intl: 1-650-362-0488
www.cloudera.com
Release Information
Version: 4.3.0
Date: May 28, 2013
Table of Contents
About this Guide ......................................................................................................15
What's New in CDH4................................................................................................17
Before You Install CDH4 on a Cluster ...................................................................19
CDH4 Installation.....................................................................................................21
CDH4 and MapReduce ..............................................................................................................................21
MapReduce 2.0 (YARN) .........................................................................................................................................21
Ways To Install CDH4 ...............................................................................................................................21
How Packaging Affects CDH4 Deployment ........................................................................................................22
Before You Begin Installing CDH4 Manually...........................................................................................22
Installing CDH4..........................................................................................................................................23
Step 1: Add or Build the CDH4 Repository or Download the "1-click Install" package. ..................................23
Step 1a: Optionally Add a Repository Key............................................................................................................26
Step 2: Install CDH4 with MRv1.............................................................................................................................26
Step 3: Install CDH4 with YARN.............................................................................................................................28
Step 4: (Optional) Install LZO.................................................................................................................................30
Step 5: Deploy CDH and Install Components.......................................................................................................31
Installing CDH4 Components...................................................................................................................31
Viewing the Apache Hadoop Documentation.........................................................................................32
Installing an Earlier CDH4 Release........................................................................33
Downloading and Installing an Earlier Release......................................................................................33
On Red Hat-compatible systems.........................................................................................................................33
On SLES systems....................................................................................................................................................34
On Ubuntu and Debian systems...........................................................................................................................35
Upgrading from CDH3 to CDH4...............................................................................37
CDH4 and MapReduce...............................................................................................................................37
MapReduce 2.0 (YARN)..........................................................................................................................................37
High Availability......................................................................................................................................................38
Before You Begin.......................................................................................................................................38
Plan Downtime.......................................................................................................................................................38
Considerations for Secure Clusters......................................................................................................................38
Upgrading to CDH4....................................................................................................................................39
Step 1: Back Up Configuration Data and Uninstall Components.......................................................................39
Step 2: Back up the HDFS Metadata.....................................................................................................................40
Step 3: Copy the Hadoop Configuration to the Correct Location and Update Alternatives............................40
Step 4: Uninstall CDH3 Hadoop.............................................................................................................................41
Step 5: Download CDH4..........................................................................................................................................42
Step 6a: Install CDH4 with MRv1..........................................................................................................................44
Step 6b: Install CDH4 with YARN..........................................................................................................................45
Step 7: Copy the CDH4 Logging File......................................................................................................................46
Step 7a: (Secure Clusters Only) Set Variables for Secure DataNodes...............................................................47
Step 8: Upgrade the HDFS Metadata....................................................................................................................47
Step 9: Create the HDFS /tmp Directory..............................................................................................................48
Step 10: Start MapReduce (MRv1) or YARN.........................................................................................................48
Step 11: Set the Sticky Bit......................................................................................................................................52
Step 12: Re-Install CDH4 Components.................................................................................................................52
Step 13: Apply Configuration File Changes..........................................................................................................52
Step 14: Finalize the HDFS Metadata Upgrade...................................................................................................53
Migrating data between a CDH3 and CDH4 cluster.............................................55
Requirements............................................................................................................................................55
Using DistCp to Migrate Data between two Clusters............................................................................55
The DistCp Command.............................................................................................................................................55
Post-migration Verification......................................................................................................................56
Upgrading from an Earlier CDH4 Release ............................................................57
Before You Begin.......................................................................................................................................57
Upgrading to the Latest Version of CDH4...............................................................................................58
Step 1: Prepare the cluster for the upgrade........................................................................................................58
Step 2: Download the CDH4 package on each of the hosts in your cluster.....................................................59
Step 3: Upgrade the packages on the appropriate hosts...................................................................................61
Step 4: Upgrade the HDFS Metadata (Beta 1 or earlier).....................................................................................64
Step 5: Start HDFS (Beta 2 or later).......................................................................................................................65
Step 5a: Verify that /tmp Exists and Has the Right Permissions.....................................................................65
Step 6: Start MapReduce (MRv1) or YARN...........................................................................................................66
Step 7: Set the Sticky Bit........................................................................................................................................70
Step 8: Upgrade Components to CDH4.................................................................................................................71
Step 9: Apply Configuration File Changes............................................................................................................71
Step 10: Finalize the HDFS Metadata Upgrade (Beta 1 or earlier)....................................................................71
Configuring Ports for CDH4.....................................................................................73
Ports Used by Components of CDH4.......................................................................................................73
Ports Used by Third Parties......................................................................................................................77
Deploying CDH4 in Pseudo-Distributed Mode.....................................................79
Deploying CDH4 on a Cluster .................................................................................81
Configuring Network Names....................................................................................................................81
Deploying HDFS on a Cluster ...................................................................................................................82
Copying the Hadoop Configuration ......................................................................................................................82
Customizing Configuration Files ..........................................................................................................................83
Configuring Local Storage Directories .................................................................................................................84
Configuring DataNodes to Tolerate Local Storage Directory Failure ...............................................................86
Formatting the NameNode ..................................................................................................................................86
Configuring a Remote NameNode Storage Directory ........................................................................................87
Configuring the Secondary NameNode ...............................................................................................................87
Enabling Trash .......................................................................................................................................................89
Configuring Storage-Balancing for the DataNodes............................................................................................90
Enabling WebHDFS ................................................................................................................................................90
Configuring LZO .....................................................................................................................................................90
Deploy MRv1 or YARN ...........................................................................................................................................91
Deploying MapReduce v1 (MRv1) on a Cluster.......................................................................................91
Step 1: Configuring Properties for MRv1 Clusters..............................................................................................92
Step 2: Configure Local Storage Directories for Use by MRv1 Daemons..........................................................92
Step 3: Configure a Health Check Script for DataNode Processes....................................................................93
Step 4: Configure JobTracker Recovery.................................................................................................................94
Enabling JobTracker Recovery...............................................................................................................................94
Step 5: Deploy your Custom Configuration to your Entire Cluster....................................................................94
Step 6: Start HDFS on Every Node in the Cluster................................................................................................95
Step 7: Create the HDFS /tmp Directory..............................................................................................................95
Step 8: Create MapReduce /var directories.........................................................................................................95
Step 9: Verify the HDFS File Structure..................................................................................................................95
Step 10: Create and Configure the mapred.system.dir Directory in HDFS.......................................................95
Step 11: Start MapReduce.....................................................................................................................................96
Step 12: Create a Home Directory for each MapReduce User............................................................................96
Configure the Hadoop Daemons to Start at Boot Time......................................................................................96
Deploying MapReduce v2 (YARN) on a Cluster.......................................................................................96
About MapReduce v2 (YARN)................................................................................................................................97
Step 1: Configure Properties for YARN Clusters..................................................................................................97
Step 2: Configure YARN daemons.........................................................................................................................98
Step 3: Configure the History Server..................................................................................................................101
Step 4: Configure the Staging Directory.............................................................................................................101
Step 5: Deploy your custom Configuration to your Entire Cluster..................................................................102
Step 6: Start HDFS on Every Node in the Cluster..............................................................................................102
Step 7: Create the HDFS /tmp Directory............................................................................................................102
剩余293页未读,继续阅读
资源评论
- kingfang0072015-02-24还可以,参考一下
- 奔跑吧小鸟2014-04-07有用,初次搭建,谢谢
- chenyaodian1232015-03-20比较好的CLOUDERA学习资料,值得推荐
ace-wangqi
- 粉丝: 1
- 资源: 1
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功