为Apache Hadoop、Spark以及Tez等大数据计算框架集成.zip

# HADOOP-COS ## 功能说明 Hadoop-COS实现了以腾讯云 COS 作为底层文件系统运行上层计算任务的功能，支持使用Hadoop、Spark以及Tez等处理存储在腾讯云COS对象存储系统上的数据。 ## 使用限制 只适用于 COS V5 版本 ## 使用环境 ### 系统环境 Linux 或 Windows 系统 ### 软件依赖 Hadoop-2.6.0及以上版本 **NOTE**： 1. 目前hadoop-cos已经正式被Apache Hadoop-3.3.0官方集成：[https://hadoop.apache.org/docs/r3.3.0/hadoop-cos/cloud-storage/index.html](https://hadoop.apache.org/docs/r3.3.0/hadoop-cos/cloud-storage/index.html)。 2. 在Apache Hadoop-3.3.0 之前版本或CDH集成Hadoop-cos jar 包后，需要重启NameNode才能加载到jar包。 3. 需要编译具体Hadoop版本的jar包可更改pom文件中hadoop.version进行编译。 ## 安装方法 ### 获取 hadoop-cos 分发包及其依赖 下载地址：[hadoop-cos release](https://github.com/tencentyun/hadoop-cos/releases) ### 安装hadoop-cos 1. 将hadoop-cos-{hadoop.version}-x.x.x.jar和cos_api-bundle-5.x.x.jar 拷贝到 `$HADOOP_HOME/share/hadoop/tools/lib`下。 NOTE: 根据hadoop的具体版本选择对应的jar包，若release中没有提供匹配版本的jar包，可自行通过修改pom文件中hadoop版本号，重新编译生成。 2. 修改 hadoop_env.sh 在 `$HADOOP_HOME/etc/hadoop`目录下，进入 hadoop_env.sh，增加如下内容，将 cosn 相关 jar 包加入 Hadoop 环境变量： ```shell for f in $HADOOP_HOME/share/hadoop/tools/lib/*.jar; do if [ "$HADOOP_CLASSPATH" ]; then export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f else export HADOOP_CLASSPATH=$f fi done ``` ## 使用方法 ### HADOOP配置 修改 $HADOOP_HOME/etc/hadoop/core-site.xml，增加 COS 相关用户和实现类信息，例如： ```xml <configuration> <property> <name>fs.defaultFS</name> <value>cosn://examplebucket-1250000000000</value> </property> <property> <name>fs.cosn.credentials.provider</name> <value>org.apache.hadoop.fs.auth.SimpleCredentialProvider</value> <description> This option allows the user to specify how to get the credentials. Comma-separated class names of credential provider classes which implement com.qcloud.cos.auth.COSCredentialsProvider: 1.org.apache.hadoop.fs.auth.SessionCredentialProvider: Obtain the secretId and secretKey from the URI:cosn://secretId:secretKey@example-1250000000000/; 2.org.apache.hadoop.fs.auth.SimpleCredentialProvider: Obtain the secret id and secret key from fs.cosn.userinfo.secretId and fs.cosn.userinfo.secretKey in core-site.xml; 3.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider: Obtain the secret id and secret key from system environment variables named COS_SECRET_ID and COS_SECRET_KEY. If unspecified, the default order of credential providers is: 1. org.apache.hadoop.fs.auth.SessionCredentialProvider 2. org.apache.hadoop.fs.auth.SimpleCredentialProvider 3. org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider 4. org.apache.hadoop.fs.auth.SessionTokenCredentialProvider 5. org.apache.hadoop.fs.auth.CVMInstanceCredentialsProvider 6. org.apache.hadoop.fs.auth.CPMInstanceCredentialsProvider 7. org.apache.hadoop.fs.auth.EMRInstanceCredentialsProvider </description> </property> <property> <name>fs.cosn.userinfo.secretId</name> <value>xxxxxxxxxxxxxxxxxxxxxxxxx</value> <description>Tencent Cloud Secret Id</description> </property> <property> <name>fs.cosn.userinfo.secretKey</name> <value>xxxxxxxxxxxxxxxxxxxxxxxx</value> <description>Tencent Cloud Secret Key</description> </property> <property> <name>fs.cosn.bucket.region</name> <value>ap-xxx</value> <description>The region where the bucket is located</description> </property> <property> <name>fs.cosn.bucket.endpoint_suffix</name> <value>cos.ap-xxx.myqcloud.com</value> <description>COS endpoint to connect to. For public cloud users, it is recommended not to set this option, and only the correct area field is required.</description> </property> <property> <name>fs.cosn.impl</name> <value>org.apache.hadoop.fs.CosFileSystem</value> <description>The implementation class of the CosN Filesystem</description> </property> <property> <name>fs.AbstractFileSystem.cosn.impl</name> <value>org.apache.hadoop.fs.CosN</value> <description>The implementation class of the CosN AbstractFileSystem.</description> </property> <property> <name>fs.cosn.tmp.dir</name> <value>/tmp/hadoop_cos</value> <description>Temporary files will be placed here.</description> </property> <property> <name>fs.cosn.upload.buffer</name> <value>mapped_disk</value> <description>The type of upload buffer. Available values: non_direct_memory, direct_memory, mapped_disk</description> </property> <property> <name>fs.cosn.upload.buffer.size</name> <value>33554432</value> <description>The total size of the upload buffer pool. -1 means unlimited.</description> </property> <property> <name>fs.cosn.upload.part.size</name> <value>8388608</value> <description>The part size for MultipartUpload. Considering the COS supports up to 10000 blocks, user should estimate the maximum size of a single file. For example, 8MB part size can allow writing a 78GB single file.</description> </property> <property> <name>fs.cosn.maxRetries</name> <value>3</value> <description>The maximum number of retries for reading or writing files to COS, before we signal failure to the application.</description> </property> <property> <name>fs.cosn.retry.interval.seconds</name> <value>3</value> <description>The number of seconds to sleep between each COS retry.</description> </property> <property> <name>fs.cosn.read.ahead.block.size</name> <value>‭1048576‬</value> <description> Bytes to read ahead during a seek() before closing and re-opening the cosn HTTP connection. </description> </property> <property> <name>fs.cosn.read.ahead.queue.size</name> <value>8</value> <description>The length of the pre-read queue.</description> </property> <property> <name>fs.cosn.customer.domain</name> <value></value> <description>The customer domain.</description> </property> <property> <name>fs.cosn.server-side-encryption.algorithm</name> <value></value> <description>The server side encryption algorithm.</description> </property> <property> <name>fs.cosn.server-side-encryption.key</name> <value></value> <description>The SSE-C server side encryption key.</description> </property> <property> <name>fs.cosn.client-side-encryption.enabled</name> <value></value> <description>Enable or disable the client encryption function</description> </property> <property> <name>fs.cosn.client-side-encryption.public.key.path</name> <value>/xxx/xxx.key</value> <description>The absolute path to the public key</description> </property> <property> <name>fs.cosn.client-side-encryption.private.key.path</name> <value>/xxx/xxx.key</value> <description>The absolute path to the private key</description> </property> </configuration> ``` **配置项说明**： | 属性键 | 说明