Apache IoTDB 是针对时间序列数据收集、存储与分析一体化的数据管理引擎 它具有体量轻、性能高、易使用的特点.zip

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # TsFile-Spark-Connector User Guide ## 1. About TsFile-Spark-Connector TsFile-Spark-Connector implements the support of Spark for external data sources of Tsfile type. This enables users to read, write and query Tsfile by Spark. With this connector, you can * load a single TsFile, from either the local file system or hdfs, into Spark * load all files in a specific directory, from either the local file system or hdfs, into Spark * write data from Spark into TsFile ## 2. System Requirements |Spark Version | Scala Version | Java Version | TsFile | |------------- | ------------- | ------------ |------------ | | `>= 2.2` | `2.11` | `1.8` | `0.10.0`| > Note: For more information about how to download and use TsFile, please see the following link: https://github.com/apache/incubator-iotdb/tree/master/tsfile. ## 3. Quick Start ### Local Mode Start Spark with TsFile-Spark-Connector in local mode: ``` ./<spark-shell-path> --jars tsfile-spark-connector.jar,tsfile-0.10.0-jar-with-dependencies.jar ``` Note: * \<spark-shell-path> is the real path of your spark-shell. * Multiple jar packages are separated by commas without any spaces. * See https://github.com/apache/iotdb/tree/master/tsfile for how to get TsFile. ### Distributed Mode Start Spark with TsFile-Spark-Connector in distributed mode (That is, the spark cluster is connected by spark-shell): ``` . /<spark-shell-path> --jars tsfile-spark-connector.jar,tsfile-0.10.0-jar-with-dependencies.jar --master spark://ip:7077 ``` Note: * \<spark-shell-path> is the real path of your spark-shell. * Multiple jar packages are separated by commas without any spaces. * See https://github.com/apache/iotdb/tree/master/tsfile for how to get TsFile. ## 4. Data Type Correspondence | TsFile data type | SparkSQL data type| | --------------| -------------- | | BOOLEAN | BooleanType | | INT32 | IntegerType | | INT64 | LongType | | FLOAT | FloatType | | DOUBLE | DoubleType | | TEXT | StringType | ## 5. Schema Inference The way to display TsFile is dependent on the schema. Take the following TsFile structure as an example: There are three Measurements in the TsFile schema: status, temperature, and hardware. The basic information of these three measurements is as follows: | name | type | encode| |------|------|-------| | status | Boolean | PLAIN| | temperature | Float | RLE| | hardware | Text | PLAIN| The existing data in the TsFile is as follows: | root.ln.wf01.wt01 | | root.ln.wf02.wt02 | | | | | | |------|------------|-----|--------|------|-------|------|-------| | status | | temperature | | hardware | | status | | | time | value | time | value | time | value | | 1 | True | 1 | 2.2 | 2 | "aaa" | 1 | True | 3 | True | 2 | 2.2 | 4 | "bbb" | 2 | False | 5 | False | 3 | 2.1 | 6 | "ccc" | 4 | True The corresponding SparkSQL table is as follows: | time | root.ln.wf02.wt02.temperature | root.ln.wf02.wt02.status | root.ln.wf02.wt02.hardware | root.ln.wf01.wt01.temperature | root.ln.wf01.wt01.status | root.ln.wf01.wt01.hardware | |------|-------------------------------|--------------------------|----------------------------|-------------------------------|--------------------------|----------------------------| | 1 | null | true | null | 2.2 | true | null | | 2 | null | false | aaa | 2.2 | null | null | | 3 | null | null | null | 2.1 | true | null | | 4 | null | true | bbb | null | null | null | | 5 | null | null | null | null | false | null | | 6 | null | null | ccc | null | null | null | You can also use narrow table form which as follows: (You can see part 6 about how to use narrow form) | time | device_name | status | hardware | temperature | |------|-------------------------------|--------------------------|----------------------------|-------------------------------| | 1 | root.ln.wf02.wt01 | true | null | 2.2 | | 1 | root.ln.wf02.wt02 | true | null | null | | 2 | root.ln.wf02.wt01 | null | null | 2.2 | | 2 | root.ln.wf02.wt02 | false | aaa | null | | 3 | root.ln.wf02.wt01 | true | null | 2.1 | | 4 | root.ln.wf02.wt02 | true | bbb | null | | 5 | root.ln.wf02.wt01 | false | null | null | | 6 | root.ln.wf02.wt02 | null | ccc | null | ## 6. Scala API NOTE: Remember to assign necessary read and write permissions in advance. ### Example 1: read from the local file system ```scala import org.apache.iotdb.tsfile._ val wide_df = spark.read.tsfile("test.tsfile") wide_df.show val narrow_df = spark.read.tsfile("test.tsfile", true) narrow_df.show ``` ### Example 2: read from the hadoop file system ```scala import org.apache.iotdb.tsfile._ val wide_df = spark.read.tsfile("hdfs://localhost:9000/test.tsfile") wide_df.show val narrow_df = spark.read.tsfile("hdfs://localhost:9000/test.tsfile", true) narrow_df.show ``` ### Example 3: read from a specific directory ```scala import org.apache.iotdb.tsfile._ val df = spark.read.tsfile("hdfs://localhost:9000/usr/hadoop") df.show ``` Note 1: Global time ordering of all TsFiles in a directory is not supported now. Note 2: Measurements of the same name should have the same schema. ### Example 4: query in wide form ```scala import org.apache.iotdb.tsfile._ val df = spark.read.tsfile("hdfs://localhost:9000/test.tsfile") df.createOrReplaceTempView("tsfile_table") val newDf = spark.sql("select * from tsfile_table where `device_1.sensor_1`>0 and `device_1.senso