centos7下kafka和zookeeper和Redis集群搭建.zip

http://blog.csdn.net/gaowenhui2008/article/details/71516901 https://cwiki.apache.org/confluence/display/KAFKA/Clients centos7下kafka集群和zookeeper集群搭建： ------------------------------------------------------------------------------------------------------------ zookeeper集群搭建：zk集群：10.1.11.177:2181,10.1.11.178:2181,10.1.11.179:2181 在/usr/local目录下新建zookeeper目录，在zookeeper下新建zookeeper_dataDir目录 将压缩包上传至zookeeper目录 cd /usr/local/zookeeper/ tar -zxvf zookeeper-3.4.8.tar.gz 进入到/usr/local/zookeeper/zookeeper-3.4.8/config目录 修改zoo.cfg 在/usr/local/zookeeper_dataDir 目录下 新建 myid 文件（里边的id编号与server.1 2 3对应） server.1=10.1.11.177:2888:3888 server.2=10.1.11.178:2888:3888 server.3=10.1.11.179:2888:3888 vim /etc/profile vi /etc/profile # Zookeeper export ZOOKEEPER_HOME=/usr/local/zookeeper/zookeeper-3.4.8 export PATH=$ZOOKEEPER_HOME/bin:$PATH # java export JAVA_HOME=/usr/local/java/jdk/jdk1.8.0_131 export JRE_HOME=/usr/local/java/jdk/jdk1.8.0_131/jre export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH source /etc/profile vim /etc/hosts cd /usr/local/zookeeper/zookeeper-3.4.8/bin ./zkServer.sh start ./zkServer.sh status ./zkServer.sh restart ./zkServer.sh stop 查看防火墙状态的命令： 　　firewall-cmd --state 关闭防火墙的命令： 　　systemctl stop firewalld.service 　　systemctl disable firewalld.service （禁止开机启动，永久关闭防火墙） ------------------------------------------------------------------------------------------------------------------------------ 搭建kafka集群： 在/usr/local目录下新建kafka目录，将压缩包上传至kafka目录 #解压软件 tar -zxvf kafka_2.11-0.9.0.1.tgz cd /usr/local/kafka/kafka_2.11-0.9.0.1/bin 修改配置文件 进入到config目录 cd /usr/local/kafka/kafka_2.11-0.9.0.1/config/ 主要关注：server.properties (broker.id , port, host.name, zookeeper.connect) broker.id=0 port=9092 host.name=10.1.11.177 broker.id=1 port=9093 host.name=10.1.11.178 broker.id=2 port=9094 host.name=10.1.11.179 启动kafka cd /usr/local/kafka/kafka_2.11-0.9.0.1/bin ./kafka-server-start.sh -daemon ../config/server.properties 1>/dev/null 2>&1 & 检查服务是否启动 jps 创建一个topic ./kafka-topics.sh --create --zookeeper 10.1.11.177:2181 --replication-factor 2 --partitions 1 --topic message-topic ./kafka-topics.sh --create --zookeeper 10.1.11.177:2181,10.1.11.178:2181,10.1.11.179:2181 --replication-factor 1 --partitions 1 --topic hello-topic ./kafka-topics.sh --create --zookeeper 10.1.11.177:2181 --replication-factor 2 --partitions 1 --topic push ./kafka-topics.sh --create --zookeeper 10.1.11.177:2181 --replication-factor 2 --partitions 1 --topic message-topic #解释 --replication-factor 2 #复制两份 --partitions 1 #创建1个分区 --topic #主题为hello topic列表 ./kafka-topics.sh --list --zookeeper 10.1.11.177:2181,10.1.11.178:2181,10.1.11.179:2181 ./kafka-topics.sh --list --zookeeper 10.1.8.99:2181 ./kafka-topics.sh --list --zookeeper 10.1.8.151:2181 ./kafka-topics.sh --list --zookeeper 10.1.11.177:2181 ./kafka-topics.sh --list --zookeeper 10.1.11.178:2181 ./kafka-topics.sh --list --zookeeper 10.1.11.179:2181 查看topic状态 ./kafka-topics.sh --describe --zookeeper 10.1.11.177:2181 --topic message-topic ./kafka-topics.sh --describe --zookeeper 10.1.11.179:2181 --topic message-topic 查看集群情况： ./kafka-topics.sh --describe --zookeeper 10.1.11.177:2181 --topic my-replicated-topic ./kafka-topics.sh --describe --zookeeper 10.1.11.178:2181 --topic PUSH-TOPIC 发现都能看到hello 创建一个broker，发布者 ./kafka-console-producer.sh --broker-list 10.1.8.98:9092 --topic PUSH-TOPIC ./kafka-console-producer.sh --broker-list 10.1.8.99:9093 --topic hello 在一台服务器上创建一个订阅者 ./kafka-console-consumer.sh --zookeeper 10.1.8.99:2181 --topic message-topic --from-beginning ./kafka-console-consumer.sh --zookeeper 10.1.8.98:2181 --topic PUSH-TOPIC --from-beginning 停止 ./kafka-server-stop.sh ./kafka-server-start.sh -daemon ../config/server.properties ------------------------------Redis哨兵集群搭建（176主、178、179）-------------------------------------------------------------------------------------------- 启动10.1.11.177的redis cd /usr/local/redis/redis-3.0.5/src ./redis-server & ----------------------------------- # wget http://download.redis.io/releases/redis-3.0.5.tar.gz # yum install gcc tcl # tar -zxf redis-3.0.5.tar.gz # cd redis-3.0.5 # make # cp /usr/local/java/redis-3.0.5/redis.conf /usr/local/bin # cd src # redis-server cd /usr/local/java/redis-3.0.5 ./src/redis-server /usr/local/java/redis-3.0.5/redis.conf & ./src/redis-sentinel /usr/local/java/redis-3.0.5/sentinel.conf & 修改三个系统中的sentinel.conf port 26379 daemonize yessentinel monitor mymaster 10.1.11.176 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 10000 sentinel auth-pass mymaster 123456 protected-mode no logfile "/usr/local/java/redis-3.0.5/sentinel.log" ps -ef|grep redis connect-redis.properties 文件放到 resources目录下