使用Ambari搭建Hadoop集群

所需积分/C币:6 2018-08-20 546KB pdf
评分

Apache Ambari是一种基于Web的工具,支持Apache Hadoop集群的供应、管理和监控。Ambari已支持大多数Hadoop组件,包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。
Hive, Hbase, Scoop, Zookeeper等),而并不仅是特指 Hadoop。用一句话来说, Ambari就是为了让 Hadoop以及相关的大数据软件更容易使用的一个工具 Amba提供了对 Hadoop更加方便快捷的管理功能,主要包含 通过一步一步的安装向导简化了集群供应。 预先配置好关鍵的运维指标( metrics),可以直接查看 Hadoop Core(HDFS 和 MapReduce)及相关项目(如 HBase、Hive和 CAtalog)是否健康。 支持作业与任务执行的可视化与分析,能够更好地査看依赖和性能。 通过一个完整的 RESTful AP把监控信息暴露岀来,集成了现有的运维工具, 用户界面非常直观,用户可以轻松有效地査看信息并控制集群。 使用Amba部署 Hadoop集群 在使用 Ambari安装部署 Hadoop时,需要对下载镜像源,配置本地环境。 配置说明 主机: node-1:192.168.10.11,配置:2C8G-30G,yum镜像源数据库,java环境 node-2∶19216810.12,配置:2C8G-30G,java环境 node-2:192.168.10.12,配置:2C8G-30G,java环境 软件包 Hadoop软件镜像压缩包, Ambari镜像压缩包,版本2.6 配置本地yum源 1、在node-1上配置hadoop所有组件的yum源。安装httpd [root@node-1#yuminstallhttpd-y 2、在官方下载鏡像文件此文件大概7G,可以使用ρ2ρ工具下载,其中包含两个repo文件 和4个压缩包 wgethttp://public-repo 1. hortonworks. com/ambari/centos7/2. x/updates/2.6. 1.0/ambari repo wgethttp://public-repo- 1.hortonworks. com/MIDP/centos7/2.x/updates/2.6.4./hdp. repo wgethttp://public-repo- 1. hortonworks. com/ambari/centos7/2. x/updates/2. 6. 1.0/ambari-2 6.1.0- centos. tar. gz wgethttp://public-repo-1.hortonworks.com/iidp-utilS-1.1.0.22/repos/centos7/iidp UTILS-I.1.0. 22-centosT tar gz wgethttp://public-repo-1.hortonworks.com/hdp GPL/ centos7/2.x/updates/2. 6. 4. 0/HDP-GPL-2 64.0-centos7-rpm tar gz wgethttp://public-repo-1.hortonworks.com/hdp/centoS7/2.x/Updates/2.6.4.0/hdp- 2.6.4.0-centos7-rpm. tar. gz 3、将对应的tar包解压到httpd的文件目录 [root@node-1 html]# tar xf ambari-2 6 1.0-centos7. tar. gz [root@node-1 html]# tar xf HDP-2 6 4.0-centos7-rpm tar gz [root@node-1 html]# tar xf HDP-GPL-26 4.0-centos7-rpm tar gz Troot@node-1 html]# mkdir IIDP-UTILS [root@node-1 html]# tar xf HDP-UTILS-1.1. 0. 22-centos7 tar gz -C HDP-UTILS/ 4、配置基础源,创建 hadoop的repo文件,修改repo文件源路径 # ambari源 vim /etc/yum. repo. d/ ambari repo [ ambari-2.6.1.0 namc=ambari version- i-2.6.1.0 baseurl=http://192.168.10.11/ambari/centos7/2.6.1.0-143 gpgcheck=1 gpgkey=http://192.168.10.11/ambari/centos7/2.6.1.0-143/rpm-gpg-key/rpm-gpg-key- Jenkins enabled=1 priority #HDP源: vim /etc/yum. repo. d/hdp repo +VErSION NUMBER-2.6.1.0-91 LHDP2.6.4.0 name=lDP Version- dP-2.6.4.0 baseurl=http://192.168.10.11/hdp/cenTos7/2.6.4.0-91 gpgcheck-1 gpgkey=http://192.168.10.11/hdp/ceNtoS7/2.6.4.0-91/rpm-gpg-key/rpm-gpg-key Jenkins enabled=1 priority=1 LIIDP-UTILS-1. 1.0. 22] name=HDP-UTIls Version- HDP-LTils-110. 22 baseurl=http://192.168.10.11/hdp-uTilS gpgcheck-= gpgkey=http://192.168.10.11/iidp-utils/rpM-GpGkEy/rpm-GpgKey-jenKins enabled-1 priority=1 [ HDP-GPL-2.6.4.0] name-HDP-GPL Version - HDP-GPL-2 6.4.0 baseurl=http://192.168.10.11/hdp-gpL/ceNtoS7/2.6.4.0-91 gpgcheck=1 gpgkey=http://192.168.10.11/hdp-gpl/CentOs7/2.6.4.0-91/rpM-gpg-key/rpm-gpg-key- skins enabled=1 priority 启动httpd 5、将本地源的repo配置拷贝到其它节点,并创建缓存 [root@node-1# scp/etc/yum. repos. d/ ambari repo 192.168.10.12:/etc/yum. repos.d [root@node-1# scp/etc/yum. repos. d/ ambari repo 192.168.10.13:/etc/yum. repos.d/ [root@node-1# scp /etc/yum. repos. d, hdp repo 192. 168 10.12: /etc/yum. repos. d/ [root@node-1# scp /etc/yum. repos. d/hdp repo 192. 168. 10.13: /etc/yum. repos. d/ 在各个节点创建缓存 t yum clean all yum makecache fast 初始化环境 各个节点安装ava-180- openjdk: yum insta.ll java-1.8.0-open jdk 2、解析主机名 echo192 168 10 11 node-1>>/etc/hosts echo 192. 168. 10. 12 node-2">>/etc/hosts echo192 168 10.13 node-3>>/etc/hosts 3、创建主机信任关系,主要是 master节点到save节点: [root(@node-1]# ssh-keygen -t rsa Root@node-1]# cp id rsa. pub/root/ ssh/authorized kcys [root@node-1]# scp id rsa. pub 192. 168.10 12: /root/ ssh/authorized keys [root@node-1]# scp id rsa. pub 192. 168 10.13: /root/ ssh/authorized keys 4、安装配置数据库: yum install mariadb-server -y systemctl start mariadb mysql secure installation 牛创建数据库: MariaDb [(none)]> create database ambari default character set utf8 Query 0K, 1 row affected(0. 00 sec) MariaDb [(none)]> grant all on ambari. *k to ambari@localhost identified by biodata Query OK, 0 rows affected (0. 00 sec MariaDB [(none)]> grant all on ambari. to ambari @% identified by ' bigdata Query oK, 0 rows affccted (0.00 scc MariaDb [(none)]> create database hive default character set utf8 Query OK, 1 row affected (0. 00 sec MariaDB [(none)]> grant all on hive. i to hive@localhost identified by ' hive Query oK, 0 rows affected (0. 00 sec Mariadb [(none)>grant all on hive. x to hive@% identified by ' hive 安装Amab服务 1、在node-1上安装 ambari- server并启动配置向导: Troot@node-1# yum install ambari-server -y Troot@node-1# ambari-scrver setup 提示如果安装配置用户时,出现如下报错 EROR: Unexpected error' getpwuido: uid not found:1001,可以查看 ambari repo文件的权限,修改为默认的root644权限即可。 2、按照配置向导信息,配置用户, java_home: Root@node-1# ambari-server setup Using python /usr/bin/ python Setup ambari-server Checking Selinux SELinux status is disabled Customize user account for ambari-server daemon [y/n](n)?y Enter user account for ambari-server daemon (root): ambari Adjusting ambari-server permissions and ownership Checking firewall status. Checking jDK [1] Oracle JDK 1.8+ Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7+ Java Cryptography Extension (JCE)Policy Files 7 3] Custom JDK Enter choice (1):3 WARNING JDK must he installed on all hosts and java home must be valid on all hosts WARNING JCe Policy files are required for configuring Kerberos security. If you plan to use Kerberos, please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts Path to JAVA HOME: /usr/lib/jvm/java-1.8.0-open jdk-1.8.0 161 0.b14.e174.x8664/jre#填写 java home Validating JdK on Ambari Server. done. Checking GPL software agreement GplLicenseforLz0:https://www.gnu.org/licenses/old-licenses/gp1-2.0.en.htm1 Enable Ambari Server to download and install GPL Licensed LZO packages [y/n n Completing setup Configuring database Enter advanced database configuration [y/n](n)? Configuring database Choose one of the following options [1]- PostgreSQL(Embedded 23-Oracle [3]- My SQL/ MariaDB [4- PostgreSQl [51-Microsoft SQL Server (Tech Preview [6]- SQL Anywhere 7J-BDB 二二二二二二二二二二二二二二二二二二二二二二二二三二二二 二二二二二二二二二二三 Enter choice(1):3 Hostname (localhost) Port(3306) Database name(ambari) Username (ambari) Enter Databasc Password(bigdata) Configuring ambari database WARNING: Before starting Ambari Server, you must copy the my sQl jdbc driver JAI file to /usr/share/java and set property server jdbc driver. path= Lpath/to/ custom jdbc driver] in ambari properties Press enter to continue 3、到上面一步时,根据提示上传mysq的jdbc驱动,并修改配置文件,指定jdbc驱动文件 位置 root@node-1# cd /usr/share/java [root @node-1 java]# 11 total 3388 rw-r--r--I root root 3467861 Jan 22 16: 16 mysql-connector-java-5 1.45. tar gz Troot@node-1 java]# tar xf mysal-connector-java-5 1. 45. tar. gz Troot@node-1 java]# mv mysql-connector-java-5. 1. 45/ mysq1-connector-java-5. 1. 45 bin. jar 修改配置文件 vim /etc/ ambari-server/conf/ ambari properties server jdbc driver. path-/usr/share/java/mysql-connector-java-5. 1. 15-bin jar 配置完成后继续,会出现如下提示: Press enter) to continue Configuring remote database connection properties WARNING: Before starting Ambari Server, you must run the following ddl against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL My SQL-CREATE sql Proceed with configuring remote database connection properties Ly/n](y) 4、岀现上述提示时,根据信息导入数据库 [root@node-1# mysql -uroot -p ambari(/var/lib/ambari server/resources/ Ambari- DdL My SQL-CREATE sq1 5、启动服务 Troot@node-1# ambari- server start 6、服务启动成功后,会监听8080端口,使用浏览器登录,账号密码 admin/ admin正常登 录,则安装完成

...展开详情
立即下载 最低0.43元/次 学生认证VIP会员7折
举报 举报 收藏 收藏
分享
img
悲催哥哥

关注 私信 TA的资源

上传资源赚积分,得勋章
相关资源标签
相关内容推荐