KRaft (aka KIP-500) mode Preview Release
=========================================================
# Introduction
It is now possible to run Apache Kafka without Apache ZooKeeper! We call this the [Kafka Raft metadata mode](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum), typically shortened to `KRaft mode`.
`KRaft` is intended to be pronounced like `craft` (as in `craftsmanship`). It is currently *PREVIEW AND SHOULD NOT BE USED IN PRODUCTION*, but it
is available for testing in the Kafka 3.1 release.
When the Kafka cluster is in KRaft mode, it does not store its metadata in ZooKeeper. In fact, you do not have to run ZooKeeper at all, because it stores its metadata in a KRaft quorum of controller nodes.
KRaft mode has many benefits -- some obvious, and some not so obvious. Clearly, it is nice to manage and configure one service rather than two services. In addition, you can now run a single process Kafka cluster.
Most important of all, KRaft mode is more scalable. We expect to be able to [support many more topics and partitions](https://www.confluent.io/kafka-summit-san-francisco-2019/kafka-needs-no-keeper/) in this mode.
# Quickstart
## Warning
KRaft mode in Kafka 3.1 is provided for testing only, *NOT* for production. We do not yet support upgrading existing ZooKeeper-based Kafka clusters into this mode.
There may be bugs, including serious ones. You should *assume that your data could be lost at any time* if you try the preview release of KRaft mode.
## Generate a cluster ID
The first step is to generate an ID for your new cluster, using the kafka-storage tool:
~~~~
$ ./bin/kafka-storage.sh random-uuid
xtzWWN4bTjitpL3kfd9s5g
~~~~
## Format Storage Directories
The next step is to format your storage directories. If you are running in single-node mode, you can do this with one command:
~~~~
$ ./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties
Formatting /tmp/kraft-combined-logs
~~~~
If you are using multiple nodes, then you should run the format command on each node. Be sure to use the same cluster ID for each one.
## Start the Kafka Server
Finally, you are ready to start the Kafka server on each node.
~~~~
$ ./bin/kafka-server-start.sh ./config/kraft/server.properties
[2021-02-26 15:37:11,071] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-02-26 15:37:11,294] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-02-26 15:37:11,466] INFO [Log partition=__cluster_metadata-0, dir=/tmp/kraft-combined-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2021-02-26 15:37:11,509] INFO [raft-expiration-reaper]: Starting (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper)
[2021-02-26 15:37:11,640] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=0, voters=[1], electionTimeoutMs=9037) (org.apache.kafka.raft.QuorumState)
...
~~~~
Just like with a ZooKeeper based broker, you can connect to port 9092 (or whatever port you configured) to perform administrative operations or produce or consume data.
~~~~
$ ./bin/kafka-topics.sh --create --topic foo --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
Created topic foo.
~~~~
# Deployment
## Controller Servers
In KRaft mode, only a small group of specially selected servers can act as controllers (unlike the ZooKeeper-based mode, where any server can become the
Controller). The specially selected controller servers will participate in the metadata quorum. Each controller server is either active, or a hot
standby for the current active controller server.
You will typically select 3 or 5 servers for this role, depending on factors like cost and the number of concurrent failures your system should withstand
without availability impact. Just like with ZooKeeper, you must keep a majority of the controllers alive in order to maintain availability. So if you have 3
controllers, you can tolerate 1 failure; with 5 controllers, you can tolerate 2 failures.
## Process Roles
Each Kafka server now has a new configuration key called `process.roles` which can have the following values:
* If `process.roles` is set to `broker`, the server acts as a broker in KRaft mode.
* If `process.roles` is set to `controller`, the server acts as a controller in KRaft mode.
* If `process.roles` is set to `broker,controller`, the server acts as both a broker and a controller in KRaft mode.
* If `process.roles` is not set at all then we are assumed to be in ZooKeeper mode. As mentioned earlier, you can't currently transition back and forth between ZooKeeper mode and KRaft mode without reformatting.
Nodes that act as both brokers and controllers are referred to as "combined" nodes. Combined nodes are simpler to operate for simple use cases and allow you to avoid
some fixed memory overheads associated with JVMs. The key disadvantage is that the controller will be less isolated from the rest of the system. For example, if activity on the broker causes an out of
memory condition, the controller part of the server is not isolated from that OOM condition.
## Quorum Voters
All nodes in the system must set the `controller.quorum.voters` configuration. This identifies the quorum controller servers that should be used. All the controllers must be enumerated.
This is similar to how, when using ZooKeeper, the `zookeeper.connect` configuration must contain all the ZooKeeper servers. Unlike with the ZooKeeper config, however, `controller.quorum.voters`
also has IDs for each node. The format is id1@host1:port1,id2@host2:port2, etc.
So if you have 10 brokers and 3 controllers named controller1, controller2, controller3, you might have the following configuration on controller1:
```
process.roles=controller
node.id=1
listeners=CONTROLLER://controller1.example.com:9093
controller.quorum.voters=1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093
```
Each broker and each controller must set `controller.quorum.voters`. Note that the node ID supplied in the `controller.quorum.voters` configuration must match that supplied to the server.
So on controller1, node.id must be set to 1, and so forth. Note that there is no requirement for controller IDs to start at 0 or 1. However, the easiest and least confusing way to allocate
node IDs is probably just to give each server a numeric ID, starting from 0.
Note that clients never need to configure `controller.quorum.voters`; only servers do.
## Kafka Storage Tool
As described above in the QuickStart section, you must use the `kafka-storage.sh` tool to generate a cluster ID for your new cluster, and then run the format command on each node before starting the node.
This is different from how Kafka has operated in the past. Previously, Kafka would format blank storage directories automatically, and also generate a new cluster UUID automatically. One reason for the change
is that auto-formatting can sometimes obscure an error condition. For example, under UNIX, if a data directory can't be mounted, it may show up as blank. In this case, auto-formatting would be the wrong thing to do.
This is particularly important for the metadata log maintained by the controller servers. If two controllers out of three controllers were able to start with blank logs, a leader might be able to be elected with
nothing in the log, which would cause all metadata to be lost.
# Missing Features
We don't support any kind of upgrade right now, either to or from KRaft mode. This is an important gap that we are working on.
Finally, the following Kafka features have not yet been fully implemented:
* Support for certain security features: configuring a KRaft-based Authorizer, setting up SCRAM, delegation tokens, and so forth
没有合适的资源?快使用搜索试试~ 我知道了~
kafka_2.13-3.1.2.tgz
0 下载量 179 浏览量
2024-06-21
22:13:04
上传
评论
收藏 84.01MB TGZ 举报
温馨提示
Kafka :最初是由 Linkedln 公司采用 Scala 语言开发的一个多分区、多副本并且基于 ZooKeeper 协调的分布式消息系统,现在已经捐献给了 Apache 基金会。目前 Kafka 已经定位为一个分布式流式处理平台,它以高吞吐、可持久化、可水平扩展、支持流处理等多种特性而被广泛应用。 Apache Kafka 是一个分布式的发布-订阅消息系统,能够支撑海量数据的数据传递。在离线和实时的消息处理业务系统中,Kafka 都有广泛的应用。Kafka 将消息持久化到磁盘中,并对消息创建了备份保证了数据的安全。Kafka 在保证了较高的处理速度的同时,又能保证数据处理的低延迟和数据的零丢失。
资源推荐
资源详情
资源评论
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![txt](https://img-home.csdnimg.cn/images/20210720083642.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![7z](https://img-home.csdnimg.cn/images/20210720083312.png)
![tgz](https://img-home.csdnimg.cn/images/20210720083646.png)
![gz](https://img-home.csdnimg.cn/images/20210720083447.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
收起资源包目录
![package](https://csdnimg.cn/release/downloadcmsfe/public/img/package.f3fc750b.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/JAR.png)
共 198 条
- 1
- 2
资源评论
![avatar-default](https://csdnimg.cn/release/downloadcmsfe/public/img/lazyLogo2.1882d7f4.png)
![avatar](https://profile-avatar.csdnimg.cn/3df60fc170844b9f87d639597dd528fb_qfyh_djh.jpg!1)
段子手-168
- 粉丝: 3535
- 资源: 2742
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![voice](https://csdnimg.cn/release/downloadcmsfe/public/img/voice.245cc511.png)
![center-task](https://csdnimg.cn/release/downloadcmsfe/public/img/center-task.c2eda91a.png)
最新资源
- Promise 前端工程化 VUE React nodejs flutter Hybrid 算法全程培训视频.zip
- MAVEN-HELPER
- inode SvpnClient
- 深度强化学习算法:DDPG TD3 SAC 实验环境:机器人MuJoCo
- stc89c52单片机C程序,串口通讯,波特兰9600,接收到指令打开继电器开关.txt
- 光伏储能+三相并离网逆变切换运行模型含笔记 包含Boost、Buck-boost双向DCDC、并网逆变器控制、离网逆变器控制
- 车辆轨迹自适应预瞄跟踪控制和自适应p反馈联合控制,自适应预苗模型和基于模糊p控制均在simulink中搭建 个人觉得跟踪效果相
- HarmonyOs 应用基础-ArkTS-核心-基础 -抽奖案例代码
- SVPWM+死区补偿(基于电流极性)+高频注入法辨识PMSM的dq轴电感(离线辨识)-simulink
- testtest0001
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![feedback](https://img-home.csdnimg.cn/images/20220527035711.png)
![feedback](https://img-home.csdnimg.cn/images/20220527035711.png)
![feedback-tip](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![dialog-icon](https://csdnimg.cn/release/downloadcmsfe/public/img/green-success.6a4acb44.png)