云雀 是一款数据集成工具，实现异构数据源的整合，帮助企业构建数据仓库、数据湖 等应用架构

# docker # 下载镜像 ```shell docker pull larkmidtable/yunque:1.0 ``` # 或者手动导入 ```shell (base) [root@lqz-test ~]# docker load -i yunque.tar fd2169e9536d: Loading layer [==================================================>] 6.144kB/6.144kB 3688a1a0ade4: Loading layer [==================================================>] 1.09MB/1.09MB 39c9d1ee8722: Loading layer [==================================================>] 3.072kB/3.072kB 7c59c0c6eb8b: Loading layer [==================================================>] 3.072kB/3.072kB 03f08cddce01: Loading layer [==================================================>] 3.584kB/3.584kB d64eeab78603: Loading layer [==================================================>] 3.584kB/3.584kB 893cebc44b3e: Loading layer [==================================================>] 3.072kB/3.072kB 96c69ca2924b: Loading layer [==================================================>] 3.072kB/3.072kB 23162ae7e027: Loading layer [==================================================>] 3.072kB/3.072kB 28f3346418e4: Loading layer [==================================================>] 102.1MB/102.1MB 4cd6816e004d: Loading layer [==================================================>] 71.74MB/71.74MB eece1ac46cba: Loading layer [==================================================>] 76.43MB/76.43MB 5f1bcb73db0e: Loading layer [==================================================>] 2.56kB/2.56kB 4cb67fbe6599: Loading layer [==================================================>] 5.632kB/5.632kB 04ad1d6a13ce: Loading layer [==================================================>] 12.29kB/12.29kB Loaded image: larkmidtable/yunque:1.0 ``` # 搭建单机测试用的kafaka docker-compose-single-broker.yml ```yaml version: '3.8' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka depends_on: [ zookeeper ] ports: - "9092:9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_CREATE_TOPICS: "test:1:1" KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 ``` # 搭建测试数据库mysql，目前支持mysql8 (如果已有数据库可跳过) ```shell docker run --restart=always \ --name test-mysql-8 \ -v /data/app/mysql8.0.21/cnf:/etc/mysql \ -v /data/app/mysql8.0.21/data:/var/lib/mysql \ -v /data/app/mysql8.0.21/log:/var/log \ -v /data/app/mysql8.0.21/mysql-files:/var/lib/mysql-files \ -p 23306:3306 \ -e MYSQL_ROOT_PASSWORD=123456 \ -d mysql:8.0.21 ``` # 构造测试配置文件 其中IP端口,数据库实例,表名,字段修改为实际使用的数据,并需要将其挂在入容器内. ```yaml reader: plugin: "mysqlreader" url: "jdbc:mysql://172.17.0.1:23306/test?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC" username: "root" password: "123456" table: "test.student" column: "id,name,address" thread: "1" writer: plugin: "mysqlwriter" url: "jdbc:mysql://172.17.0.1:23306/test?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC" username: "root" password: "123456" table: "test.student_1" column: "id,name,address" "thread": 1 kafka: host: 172.17.0.1:9092 topic: hu_topic clientId: hu_client groupId: hu_group log: logPath: /home/yunque/logs #transformer: # - {"name": "dx_digest","parameter":{ "columnName":"role_key","paras":["md5", "toLowerCase"] }} ``` # 执行测试命令 ```shell docker run --rm -it -u yunque -v /data/app/yunque/logs:/home/yunque/logs -v /data/app/yunque/test.yaml:/home/yunque/test.yaml larkmidtable/yunque:1.0 bash bin/start.sh -j test -i 1 -p /home/yunque/test.yaml -f YAML -d ``` ```text 执行启动命令 : bash bin/start.sh -j test -i 1 -p /home/yunque/test.yaml -f YAML -d test 1 /home/yunque/test.yaml YAML /home/yunque/openjdk/bin/java -server -Xms512m -Xmx512m -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/yunque/logs/java_heapdump.hprof -XX:-UseLargePages -Xlog:gc*:file=/home/yunque/logs/yunque_gc.log:time,tags:filecount=10,filesize=100m [main] INFO com.larkmidtable.yunque.YunQueEngine - Hello! 欢迎使用云雀数据集成.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 核查参数的正确性.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 核查参数的完成.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 解析传递的参数.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 作业名称test ,作业ID1 ,作业的路径/home/yunque/test.yaml ,作业文件的格式YAML [main] INFO com.larkmidtable.yunque.YunQueEngine - 读取作业配置文件.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 解析配置文件.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 加载Transformer插件.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 获取Reader和Writer.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 创建读写的线程池和计数器... [main] INFO com.larkmidtable.yunque.YunQueEngine - 进行读写任务.... [main] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader建立连接开始.... Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. [main] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader建立连接结束.... [main] INFO www.larkmidtable.com.MySQLWriter - MySQL的Writer建立连接开始.... [main] INFO www.larkmidtable.com.MySQLWriter - MySQL的Writer建立连接结束.... [main] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader开始进行分片开始.... [main] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader开始进行分片结束.... [pool-1-thread-1] INFO www.larkmidtable.com.MySQLReader - MySQL读取数据操作.... [pool-1-thread-1] INFO www.larkmidtable.com.MySQLReader - 执行的SQL:SELECT id,name,address FROM ( select id,name,address from test.student ) t LIMIT 0,1 [pool-1-thread-1] INFO www.larkmidtable.com.reader.AbstractDBReader - 添加到队列的记录条数1 [pool-1-thread-1] INFO www.larkmidtable.com.MySQLReader - MySQL读取数据结束....耗时：9ms [pool-2-thread-1] INFO www.larkmidtable.com.MySQLWriter - 开始写数据.... [pool-2-thread-1] INFO www.larkmidtable.com.MySQLWriter - 写数据完成....耗时：15ms [main] INFO com.larkmidtable.yunque.YunQueEngine - 结束迁移任务.... [main] INFO com.larkmidtable.yunque.YunQueEngine - 任务总耗时：972ms ``` ctrl+c 取消,具体执行docker 可以加 -d参数 提交到后台 ```text ^C[Thread-0] INFO www.larkmidtable.com.util.JVMUtil - 开始销毁线程池... [Thread-0] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader开始进行关闭连接开始.... [Thread-1] INFO www.larkmidtable.com.util.JVMUtil - 开始销毁线程池... [Thread-1] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader开始进行关闭连接开始.... [Thread-0] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader开始进行关闭连接结束.... [Thread-0] INFO www.larkmidtable.com.MySQLWriter - MySQL的Writer开始进行关闭连接开始.... [Thread-1] INFO www.larkmidtable.com.MySQLReader - MySQL的Reader开始进行关闭连接结束.... [Thread-1] INFO www.larkmidtable.com.MySQLWriter - MySQL的Writer开始进行关闭连接开始.... [Thread-1] INFO www.larkmidtable.com.MySQLWriter - MySQL的Writer开始进行关闭连接结束.... [Thread-1] INFO www.larkmidtable.com.util.JVMUtil - 成功销毁线程池.... [Thread-0] INFO www.larkmidtable.com.MySQLWriter - MySQL的Writer开始进行关闭连接结束.... [Thread-0] INFO www.larkmidtable.com.util.JVMUtil - 成功销毁线程池.... test start up success.... (base) [root@lqz-test ~]# ``` # 如果遇到问题需要调试 ## 1.先创建yunque的容器并进入其中 ```shell docker run --rm -it -u yunque -v /data/app/yunque/logs:/home/yunque/logs -v /data/app/y