【免费】2、sqoop导入（RMDB-mysql、sybase到HDFS-hive）_同时将数据库的数据导入到hdfs和hive资源-CSDN文库

需积分: 0 23 浏览量 2023-05-29 09:46:40 上传评论收藏 386KB PDF 举报

资源推荐

资源详情

资源评论

Apache Sqoop系列文章

1、Apache Sqoop介绍及部署

2、sqoop导入（RMDB-mysql、sybase到HDFS-hive）

3、Sqoop导出(HDFS到RMDB-mysql)

4、Sqoop job作业

@TOC

本文是在前一篇的基础上介绍sqoop的使用，具体的部署环境请参考前一篇。

本文的前提是hadoop、mysql、hive环境都是好用的。

本文分为2个部分，即sqoop导入的简单介绍以及6种使用示例，包含sybase导入到hdfs。

一、Sqoop导入介绍

“导入工具”导入单个表从RDBMS到HDFS。表中的每一行被视为HDFS的记录。

所有记录都存储为文本文件的文本数据

1、语法结构

下面的语法用于将数据导入HDFS。

2、简单验证

二、使用示例

1、全量导入mysql表数据到HDFS

前提是对应的环境都是可用的，且是有测试数据的。

1)、导入

$ sqoop import (generic-args) (import-args)

----验证sqoop连接mysql 是否成功--列出该链接下的所有库

sqoop list-databases --connect jdbc:mysql://192.168.10.44:3306/test --username

root --password 88888

# 从MySQL数据库服务器中的person表导入HDFS

sqoop import --connect jdbc:mysql://192.168.10.44:3306/test --username root --

password 888888 \

--delete-target-dir \

--target-dir /test \

--table person \

--m 1

# 其中

# --target-dir可以用来指定导出数据存放至HDFS的目录；

# --delete-target-dir  删除target-dir目录，然后再重建

[root@server7 lib]# sqoop import --connect jdbc:mysql://192.168.10.44:3306/test

--username root --password 888888 \

> --delete-target-dir \

> --target-dir /test \

> --table person \

> --m 1

Warning: /opt/cloudera/parcels/CDH-6.2.1-

1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo

imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-

1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-

1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-

1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-

2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

22/11/08 15:12:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

22/11/08 15:12:19 WARN tool.BaseSqoopTool: Setting your password on the command-

line is insecure. Consider using -P instead.

22/11/08 15:12:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming

resultset.

22/11/08 15:12:19 INFO tool.CodeGenTool: Beginning code generation

Tue Nov 08 15:12:19 CST 2022 WARN: Establishing SSL connection without server's

identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and

5.7.6+ requirements SSL connection must be established by default if explicit

option isn't set. For compliance with existing applications not using SSL the

verifyServerCertificate property is set to 'false'. You need either to

explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide

truststore for server certificate verification.

22/11/08 15:12:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.*

FROM `person` AS t LIMIT 1

22/11/08 15:12:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.*

FROM `person` AS t LIMIT 1

22/11/08 15:12:19 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is

/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce

22/11/08 15:12:22 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-

root/compile/4f96760f8620e1c038672078f7a2ed66/person.java to

/opt/cloudera/parcels/CDH-6.2.1-

1.cdh6.2.1.p0.1425774/lib/sqoop/lib/./person.java. Error: Destination

'/opt/cloudera/parcels/CDH-6.2.1-

1.cdh6.2.1.p0.1425774/lib/sqoop/lib/./person.java' already exists

22/11/08 15:12:22 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-

root/compile/4f96760f8620e1c038672078f7a2ed66/person.jar

22/11/08 15:12:23 INFO tool.ImportTool: Destination directory /test deleted.

22/11/08 15:12:23 WARN manager.MySQLManager: It looks like you are importing

from mysql.

22/11/08 15:12:23 WARN manager.MySQLManager: This transfer can be faster! Use

the --direct

22/11/08 15:12:23 WARN manager.MySQLManager: option to exercise a MySQL-specific

fast path.

22/11/08 15:12:23 INFO manager.MySQLManager: Setting zero DATETIME behavior to

convertToNull (mysql)

22/11/08 15:12:23 INFO mapreduce.ImportJobBase: Beginning import of person

22/11/08 15:12:23 INFO Configuration.deprecation: mapred.jar is deprecated.

Instead, use mapreduce.job.jar

22/11/08 15:12:23 INFO Configuration.deprecation: mapred.map.tasks is

deprecated. Instead, use mapreduce.job.maps

22/11/08 15:12:23 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to

rm46

22/11/08 15:12:23 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding

for path: /user/root/.staging/job_1667806921128_0009

Tue Nov 08 15:12:31 CST 2022 WARN: Establishing SSL connection without server's

identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and

5.7.6+ requirements SSL connection must be established by default if explicit

option isn't set. For compliance with existing applications not using SSL the

verifyServerCertificate property is set to 'false'. You need either to

explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide

truststore for server certificate verification.

22/11/08 15:12:31 INFO db.DBInputFormat: Using read commited transaction

isolation

22/11/08 15:12:31 INFO mapreduce.JobSubmitter: number of splits:1

22/11/08 15:12:31 INFO Configuration.deprecation: yarn.resourcemanager.zk-

address is deprecated. Instead, use hadoop.zk.address

22/11/08 15:12:31 INFO Configuration.deprecation: yarn.resourcemanager.system-

metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-

publisher.enabled

22/11/08 15:12:31 INFO mapreduce.JobSubmitter: Submitting tokens for job:

job_1667806921128_0009

22/11/08 15:12:31 INFO mapreduce.JobSubmitter: Executing with tokens: []

22/11/08 15:12:32 INFO conf.Configuration: resource-types.xml not found

22/11/08 15:12:32 INFO resource.ResourceUtils: Unable to find 'resource-

types.xml'.

22/11/08 15:12:32 INFO impl.YarnClientImpl: Submitted application

application_1667806921128_0009

22/11/08 15:12:32 INFO mapreduce.Job: The url to track the job:

http://server8:8088/proxy/application_1667806921128_0009/

22/11/08 15:12:32 INFO mapreduce.Job: Running job: job_1667806921128_0009

22/11/08 15:12:39 INFO mapreduce.Job: Job job_1667806921128_0009 running in uber

mode : false

22/11/08 15:12:39 INFO mapreduce.Job: map 0% reduce 0%

22/11/08 15:12:46 INFO mapreduce.Job: map 100% reduce 0%

22/11/08 15:12:46 INFO mapreduce.Job: Job job_1667806921128_0009 completed

successfully

22/11/08 15:12:47 INFO mapreduce.Job: Counters: 33

   File System Counters

       FILE: Number of bytes read=0

       FILE: Number of bytes written=248904

       FILE: Number of read operations=0

       FILE: Number of large read operations=0

       FILE: Number of write operations=0

       HDFS: Number of bytes read=85

       HDFS: Number of bytes written=248

       HDFS: Number of read operations=6

剩余11页未读，继续阅读

评论收藏

内容反馈

一瓢一瓢的饮alanchanchn

粉丝: 2827
资源: 69

2、sqoop导入（RMDB-mysql、sybase到HDFS-hive）

全分布式集群Hadoop全套组件解压即可使用包含（Ubuntu-HDFS-HBASE-HIve-Sqoop-mysql....）

sqoop导入数据到hdfs路径

3、Sqoop导出(HDFS到RMDB-mysql)

sqoop-spark-hive:MYSQL | SQOOP | 火花| HIVE工作流程

Sqoop-sqlserver-hdfs.rar

电商销售数据复盘用教程数据 hive分析-菜鸟入门

xthreg2命令安装包

Python金融量化的高级库：TA-Lib-0.4.24（包含python3.7、3.8、3.9、3.10的32位和64位版本）

2022泰迪杯数据分析技能赛B题代码 Jupyter Notebook

数据源-数据可视化（七）：Pandas香港酒店数据高级分析，涉及相关系数，协方差，数据离散化，透视表等精美可视化展示

统计数模历年优秀论文-全国大学生统计建模大赛

2022年美赛（MCM）C题M奖论文

Origin中【CorrelationPlot】插件

超强股票分析工具，财星系列

python数据分析大作业-爬虫、建立模型、可视化、结论分析

实验3客户RFM实验案例代码

中国自然保护区名录+矢量边界数据

计算方法-上机作业-示例【仅供交流参考】

天气数据爬取的源代码-可以运行

GARCH-MIDAS、DDC-MIDAS模型MATLAB代码

零基础入门转录组分析-数据处理（GEO数据库-芯片数据）配套资源

Attachment 实现 全文检索

利用citespace对知网和WOS文献做文献综述

零基础入门转录组分析-数据处理（TCGA数据库）教程配套的代码，原始数据以及最终处理好的数据

大乐透第一期至今全部期数的开奖结果（2007年-2023年5月共2423期）

zotero科研插件合集（zotero 6.0 适用）

besnew-超强二进制文件查看工具

全国流域片数据，包含一、二、三级流域划分

因果推断实用计量方法PDF

最新资源

Attachment 实现全文检索