pulsar-flink：使用Apache Pulsar和Apache Flink进行弹性数据处理

# Pulsar Flink Connector The Pulsar Flink connector implements elastic data processing using [Apache Pulsar](https://pulsar.apache.org) and [Apache Flink](https://flink.apache.org). For details about the Chinese document, see [here](doc/README_CN.md). # Prerequisites - Java 8 or higher version - Flink 1.9.0 or higher version - Pulsar 2.4.0 or higher version # Basic information This section describes basic information about the Pulsar Flink connector. ## Client Currently, the following Flink versions are supported. - Flink 1.9 - 1.10: they are maintained in the [`flink-1.9` branch](https://github.com/streamnative/pulsar-flink/tree/flink-1.9). - Flink 1.11: it is maintained in the [`flink-1.11` branch](https://github.com/streamnative/pulsar-flink/tree/flink-1.11). - Flink 1.12: it is maintained in the [`master` branch](https://github.com/streamnative/pulsar-flink/tree/master). > **Note** > Since Flink's API has changed greatly, we mainly work on new features in the `master` branch and fix bugs in other branches. The JAR package is located in the [Bintray Maven repository of StreamNative](https://dl.bintray.com/streamnative/maven). For projects using SBT, Maven, or Gradle, you can set the following parameters for your project. - `FLINK_VERSION`: currently, versions `1.9`, `1.11`, and `1.12` are available. - `SCALA_BINARY_VERSION`: this parameter defines the Scala version used by Flink. Versions `2.11` and `2.12` are available. - `PULSAR_FLINK_VERSION`: it is the version of the Pulsar Flink connector. Usually, use a three-digit version (such as version `2.7.0`) for a master release and a four-digit version for a branch release (such as version `2.7.0.1`). Here is an example about how to configure parameters for projects using SBT, Maven, or Gradle. ```shell groupId = io.streamnative.connectors artifactId = pulsar-flink-connector-{{SCALA_BINARY_VERSION}}-{{FLINK_VERSION}} version = {{PULSAR_FLINK_VERSION}} ``` ## Maven projects For Maven projects, you can add the repository configuration to your `pom.xml`, as shown below. ```xml <repositories> <repository> <id>central</id> <layout>default</layout> <url>https://repo1.maven.org/maven2</url> </repository> <repository> <id>bintray-streamnative-maven</id> <name>bintray</name> <url>https://dl.bintray.com/streamnative/maven</url> </repository> </repositories> ``` For Maven projects, you can use the following [shade](https://imperceptiblethoughts.com/shadow/) plugin definition template to build an application JAR package that contains all the dependencies required for the client library and Pulsar Flink connector. ```xml <plugin> <!-- Shade all the dependencies to avoid conflicts --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>${maven-shade-plugin.version}</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <createDependencyReducedPom>true</createDependencyReducedPom> <promoteTransitiveDependencies>true</promoteTransitiveDependencies> <minimizeJar>false</minimizeJar> <artifactSet> <includes> <include>io.streamnative.connectors:*</include> <!-- more libs to include here --> </includes> </artifactSet> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> <transformer implementation="org.apache.maven.plugins.shade.resource.PluginXmlResourceTransformer" /> </transformers> </configuration> </execution> </executions> </plugin> ``` ## Gradle projects For Gradle projects, you can add the repository configuration to your `build.gradle`, as shown below. ```groovy repositories { maven { url 'https://dl.bintray.com/streamnative/maven' } } ``` For gradle projects, you can use the following [shade](https://imperceptiblethoughts.com/shadow/) plugin definition template to build an application JAR package that contains all the dependencies required for the client library and Pulsar Flink connector. ```groovy buildscript { repositories { jcenter() } dependencies { classpath 'com.github.jengelman.gradle.plugins:shadow:6.0.0' } } apply plugin: 'com.github.johnrengelman.shadow' apply plugin: 'java' ``` # Build Pulsar Flink connector To build the Pulsar Flink connector for reading data from Pulsar or writing the results to Pulsar, follow these steps. 1. Check out the source code. ```bash git clone https://github.com/streamnative/pulsar-flink.git cd pulsar-flink ``` 2. Install the Docker. The Pulsar Flink connector uses [Testcontainers](https://www.testcontainers.org/) for integration test. To run the integration test, ensure to install the Docker. For details about how to install the Docker, see [here](https://docs.docker.com/docker-for-mac/install/). 3. Set the Java version. Modify `java.version` and `java.binary.version` in `pom.xml`. > **Note** > Ensure that the Java version should be identical to the Java version for the Pulsar Flink connector. 4. Build the project. ```bash mvn clean install -DskipTests ``` 5. Run the test. ```bash mvn clean install ``` After the Pulsar Flink connector is installed, a JAR package that contains all the dependencies is generated in both the local Maven repository and the `target` directory. > **Note** > If you use intellij IDEA to debug this project, you might encounter the `org.apache.pulsar.shade.org.bookkeeper.ledger` package error. To fix the error, use the ` mvn clean install -DskipTests` command to install the JAR package to the local repository, ignore the `managed-ledger-shaded` Maven module on the project, and then click **Refresh**. # Deploy Pulsar Flink connector This section describes how to deploy the Pulsar Flink connector. ## Client library For any Flink application, use the `./bin/flink run` command to compile and start your application. If you have already built a JAR package with dependencies using the above shade plugin, you can use the `--classpath` option to add your JAR package. > **Note** > The path must be in a protocol format (such as `file://`) and the path must be accessible on all nodes. **Example** ``` ./bin/flink run -c com.example.entry.point.ClassName file://path/to/jars/your_fat_jar.jar ``` ## Scala REPL The Scala REPL is a tool (scala) for evaluating expressions in Scala. Use the `bin/start-scala-shell.sh` command to deploy Pulsar Flink connector on Scala client. You can use the `--addclasspath` to add `pulsar-flink-connector_{{SCALA_BINARY_VERSION}}-{{ PULSAR_FLINK_VERSION}}.jar` package. **Example** ``` ./bin/start-scala-shell.sh remote <hostname> <portnumber> --addclasspath pulsar-flink-connector-{{SCALA_BINARY_VERSION}}-{{PULSAR_FLINK_VERSION}}.jar ``` For more information on submitting applications through the CLI, see [Command-Line Interface](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/cli.html) . ## SQL client The [SQL Client](https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/sqlClient.html) is used to write SQL queries for manipulating data in Pulsar, you can use the `-addclasspath` option to add `pulsar-flink-connector-{{SCALA_BINARY_VERSION}}-{{PULSAR_FLINK_VERSION}}.jar` package. **Example** ``` ./bin/sql-client.sh embedded --jar pulsar-flink-connector-{{SCALA_BINARY_VERSION}}-{{PULSAR_FLINK_VERSION}}.jar ``` > **Note** > If you put the JAR