www.aboutyun.com 活到老、学到老
about 云 Flink 1.11 中文文档
作者：pig2 整理
时间：2020.07.17
出处：about 云
www.aboutyun.com 活到老、学到老
目录
about 云 Flink 1.11 中文文档 ................................................................................................................. 1
Apache Flink 文档 ................................................................................................................................. 20
试用 Flink ........................................................................................................................................ 20
学习 Flink ........................................................................................................................................ 21
获取 Flink 帮助 ............................................................................................................................. 21
探索 Flink ........................................................................................................................................ 21
部署 Flink ........................................................................................................................................ 21
升级 Flink ........................................................................................................................................ 21
基于 DataStream API 实现欺诈检测 ............................................................................................. 22
你要搭建一个什么系统 ..................................................................................................................... 22
准备条件 ................................................................................................................................................ 22
困难求助 ................................................................................................................................................ 22
怎样跟着教程练习 .............................................................................................................................. 22
代码分析 ................................................................................................................................................ 25
实现一个真正的应用程序 ................................................................................................................ 28
欺诈检测器 v2：状态 + 时间 = .......................................................................................... 31
完整的程序............................................................................................................................................ 33
期望的结果 ....................................................................................................................................... 37
Python API 教程 ..................................................................................................................................... 37
创建一个 Python Table API 项目 ................................................................................................. 37
编写一个 Flink Python Table API 程序 ...................................................................................... 38
执行一个 Flink Python Table API 程序 ...................................................................................... 41
Table API & SQL .................................................................................................................................... 41
依赖图 ..................................................................................................................................................... 42
Table 程序依赖 .............................................................................................................................. 43
扩展依赖............................................................................................................................................ 44
接下来？ ................................................................................................................................................ 45
DataStream API 简介 .......................................................................................................................... 45
什么能被转化成流？ ......................................................................................................................... 46
Java tuples 和 POJOs ............................................................................................................... 46
www.aboutyun.com 活到老、学到老
Scala tuples 和 case classes.................................................................................................. 47
一个完整的示例 .................................................................................................................................. 47
Stream 执行环境 .......................................................................................................................... 49
基本的 stream source ................................................................................................................. 49
基本的 stream sink ...................................................................................................................... 50
调试 ..................................................................................................................................................... 50
动手实践 ................................................................................................................................................ 50
数据管道 & ETL ...................................................................................................................................... 51
无状态的转换 ....................................................................................................................................... 52
map() ................................................................................................................................................ 52
flatmap() ..................................................................................................................................... 53
Keyed Streams ................................................................................................................................... 54
keyBy()........................................................................................................................................... 54
通过计算得到键 .............................................................................................................................. 56
Keyed Stream 的聚合 ................................................................................................................. 56
（隐式的）状态 .............................................................................................................................. 57
reduce() 和其他聚合算子 ...................................................................................................... 58
有状态的转换 ....................................................................................................................................... 58
Flink 为什么要参与状态管理？ ............................................................................................... 58
Rich Functions ................................................................................................................................ 58
一个使用 Keyed State 的例子 ................................................................................................ 59
清理状态............................................................................................................................................ 61
Non-keyed State ............................................................................................................................ 61
Connected Streams .......................................................................................................................... 61
示例 ..................................................................................................................................................... 62
动手练习 ................................................................................................................................................ 64
事件驱动应用............................................................................................................................................ 64
处理函数（Process Functions） ................................................................................................. 64
简介 ..................................................................................................................................................... 64
示例 ..................................................................................................................................................... 64
性能考虑............................................................................................................................................ 69
旁路输出（Side Outputs） ............................................................................................................ 69
www.aboutyun.com 活到老、学到老
简介 ..................................................................................................................................................... 69
示例 ..................................................................................................................................................... 70
结语 ......................................................................................................................................................... 70
通过状态快照实现容错处理 ................................................................................................................ 71
State Backends .................................................................................................................................. 71
状态快照 ................................................................................................................................................ 72
定义 ..................................................................................................................................................... 72
状态快照如何工作？ ..................................................................................................................... 72
确保精确一次（exactly once） ................................................................................................ 73
端到端精确一次 .............................................................................................................................. 74
概念透析 ..................................................................................................................................................... 74
Flink 中的 API .................................................................................................................................... 74
词汇表 ......................................................................................................................................................... 75
DataStream API ........................................................................................................................................ 78
Broadcast State 模式 ........................................................................................................................... 78
提供的 API ........................................................................................................................................... 79
BroadcastProcessFunction 和 KeyedBroadcastProcessFunction .......................... 81
重要注意事项 ....................................................................................................................................... 85
Checkpointing .......................................................................................................................................... 85
前提条件 ................................................................................................................................................ 85
开启与配置 Checkpoint .................................................................................................................. 86
相关的配置选项 .............................................................................................................................. 88
选择一个 State Backend ................................................................................................................ 91
迭代作业中的状态和 checkpoint ................................................................................................. 91
重启策略 ................................................................................................................................................ 91
Queryable State Beta ........................................................................................................................... 91
架构 ......................................................................................................................................................... 92
激活 Queryable State ...................................................................................................................... 92
将 state 设置为可查询的 ............................................................................................................... 92
Queryable State Stream ............................................................................................................. 93
Managed Keyed State................................................................................................................. 94
查询 state ............................................................................................................................................. 94
www.aboutyun.com 活到老、学到老
示例 ..................................................................................................................................................... 95
Configuration ....................................................................................................................................... 97
State Server .................................................................................................................................... 98
Proxy .................................................................................................................................................. 98
限制 ......................................................................................................................................................... 98
State Backends ....................................................................................................................................... 98
状态数据结构升级 .................................................................................................................................. 99
升级状态数据结构 .............................................................................................................................. 99
数据结构升级支持的数据类型 ..................................................................................................... 100
POJO 类型 .................................................................................................................................... 100
Avro 类型 ....................................................................................................................................... 100
用于外部数据访问的异步 I/O .......................................................................................................... 101
对于异步 I/O 操作的需求 ............................................................................................................ 101
先决条件 .............................................................................................................................................. 102
异步 I/O API ...................................................................................................................................... 102
超时处理.......................................................................................................................................... 104
结果的顺序 ..................................................................................................................................... 104
事件时间.......................................................................................................................................... 105
容错保证.......................................................................................................................................... 105
实现提示.......................................................................................................................................... 105
警告 ................................................................................................................................................... 106
Java Lambda 表达式 ......................................................................................................................... 106
示例和限制 ..................................................................................................................................... 106
DataSet API .............................................................................................................................................. 110
Batch 示例 .............................................................................................................................................. 110
运行一个示例 ..................................................................................................................................... 110
Word Count ........................................................................................................................................ 111
Page Rank .......................................................................................................................................... 112
Connected Components（连通组件算法） .......................................................................... 115
Table API & SQL ..................................................................................................................................... 119
依赖图 ................................................................................................................................................... 119
Table 程序依赖 ............................................................................................................................ 120
