Apache Parquet Java.zip

<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, ~ software distributed under the License is distributed on an ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY ~ KIND, either express or implied. See the License for the ~ specific language governing permissions and limitations ~ under the License. --> # Hadoop integration **Todo:** Add a description ## Properties It's possible to configure the [ParquetInputFormat](https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java) / [ParquetOutputFormat](https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetOutputFormat.java) with Hadoop config or programmatically with setters. **Example:** ```java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapreduce.Job; Configuration conf = new Configuration(); conf.set("parquet.page.size","128"); Job writeJob = new Job(conf); ParquetOutputFormat.setBlockSize(writeJob, 1024); ``` ## Class: ParquetOutputFormat **Property:** `parquet.summary.metadata.level` **Description:** Write summary files in the same directory as parquet files. If this property is set to `all`, write both summary file with row group info to _metadata and summary file without row group info to _common_metadata. If it is `common_only`, write only the summary file without the row group info to _common_metadata. If it is `none`, don't write summary files. **Default value:** `all` --- **Property:** `parquet.enable.summary-metadata` **Description:** This property is deprecated, use `parquet.summary.metadata.level` instead. If it is `true`, it similar to `parquet.summary.metadata.level` with `all`. If it is `false`, it is similar to `NONE`. **Default value:** `true` --- **Property:** `parquet.block.size` **Description:** The block size in bytes. This property depends on the file system: - If the file system (FS) used supports blocks like HDFS, the block size will be the maximum between the default block size of FS and this property. And the row group size will be equal to this property. - block_size = max(default_fs_block_size, parquet.block.size) - row_group_size = parquet.block.size` - If the file system used doesn't support blocks, then this property will define the row group size. Note that larger values of row group size will improve the IO when reading but consume more memory when writing **Default value:** `134217728` (128 MB) --- **Property:** `parquet.page.size` **Description:** The page size in bytes is for compression. When reading, each page can be decompressed independently. A block is composed of pages. The page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. **Default value:** `1048576` (1 MB) --- **Property:** `parquet.compression` **Description:** The compression algorithm used to compress pages. This property supersedes `mapred.output.compress*`. It can be `uncompressed`, `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` and `lz4_raw`. If `parquet.compression` is not set, the following properties are checked: * mapred.output.compress=true * mapred.output.compression.codec=org.apache.hadoop.io.compress.SomeCodec Note that custom codecs are explicitly disallowed. Only one of Snappy, GZip, LZO, LZ4, Brotli or ZSTD is accepted. **Default value:** `uncompressed` --- **Property:** `parquet.write.support.class` **Description:** The write support class to convert the records written to the OutputFormat into the events accepted by the record consumer. Usually provided by a specific ParquetOutputFormat subclass and it should be the descendant class of `org.apache.parquet.hadoop.api.WriteSupport` --- **Property:** `parquet.enable.dictionary` **Description:** Whether to enable or disable dictionary encoding. If it is true, then the dictionary encoding is enabled for all columns. If it is false, then the dictionary encoding is disabled for all columns. It is possible to enable or disable the encoding for some columns by specifying the column name in the property. Note that all configurations of this property will be combined (See the following example). **Default value:** `true` **Example:** ```java // Enable dictionary encoding for all columns conf.set("parquet.enable.dictionary", true); // Disable dictionary encoding for 'column.path' conf.set("parquet.enable.dictionary#column.path", false); // The final configuration will be: Enable dictionary encoding for all columns except 'column.path' ``` --- **Property:** `parquet.dictionary.page.size` **Description:** The dictionary page size works like the page size but for dictionary. There is one dictionary page per column per row group when dictionary encoding is used. **Default value:** `1048576` (1 MB) --- **Property:** `parquet.validation` **Description:** Whether to turn on validation using the schema. **Default value:** `false` --- **Property:** `parquet.writer.version` **Description:** The writer version. It can be either `PARQUET_1_0` or `PARQUET_2_0`. `PARQUET_1_0` and `PARQUET_2_0` refer to DataPageHeaderV1 and DataPageHeaderV2. The v2 pages store levels uncompressed while v1 pages compress levels with the data. For more details, see the [thrift definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift). **Default value:** `PARQUET_1_0` --- **Property:** `parquet.memory.pool.ratio` **Description:** The memory manager balances the allocation size of each Parquet writer by resize them averagely. If the sum of each writer's allocation size is less than the total memory pool, the memory manager keeps their original value. If the sum exceeds, it decreases the allocation size of each writer by this ratio. This property should be between 0 and 1. **Default value:** `0.95` --- **Property:** `parquet.memory.min.chunk.size` **Description:** The minimum allocation size in byte per Parquet writer. If the allocation size is less than the minimum, the memory manager will fail with an exception. **Default value:** `1048576` (1 MB) --- **Property:** `parquet.writer.max-padding` **Description:** The maximum size in bytes allowed as padding to align row groups. This is also the minimum size of a row group. **Default value:** `8388608` (8 MB) --- **Property:** `parquet.page.size.row.check.min` **Description:** The frequency of checks of the page size limit will be between `parquet.page.size.row.check.min` and `parquet.page.size.row.check.max`. If the frequency is high, the page size will be accurate. If the frequency is low, the performance will be better. **Default value:** `100` --- **Property:** `parquet.page.size.row.check.max` **Description:** The frequency of checks of the page size limit will be between `parquet.page.size.row.check.min` and `parquet.page.size.row.check.max`. If the frequency is high, the page size will be accurate. If the frequency is low, the performance will be better. **Default value:** `10000` --- **Property:** `parquet.page.value.count.threshold` **Description:** The value count threshold within a Parquet page used on each page check. **Default value:** `Integer.MAX_VALUE / 2` --- **Property:** `parquet.page.size.check.estimate` **Description:** If it is true, the column writer estimat