高性能多线程数据处理库

<h1 align="center"> <img src="https://raw.githubusercontent.com/pola-rs/polars-static/master/logos/polars_github_logo_rect_dark_name.svg" alt="Polars logo"> <br> </h1> <div align="center"> <a href="https://crates.io/crates/polars"> <img src="https://img.shields.io/crates/v/polars.svg" alt="crates.io Latest Release"/> </a> <a href="https://pypi.org/project/polars/"> <img src="https://img.shields.io/pypi/v/polars.svg" alt="PyPi Latest Release"/> </a> <a href="https://www.npmjs.com/package/nodejs-polars"> <img src="https://img.shields.io/npm/v/nodejs-polars.svg" alt="NPM Latest Release"/> </a> <a href="https://rpolars.r-universe.dev"> <img src="https://rpolars.r-universe.dev/badges/polars" alt="R-universe Latest Release"/> </a> <a href="https://doi.org/10.5281/zenodo.7697217"> <img src="https://zenodo.org/badge/DOI/10.5281/zenodo.7697217.svg" alt="DOI Latest Release"/> </a> </div> <p align="center"> <b>Documentation</b>: <a href="https://docs.pola.rs/py-polars/html/reference/index.html">Python</a> - <a href="https://docs.rs/polars/latest/polars/">Rust</a> - <a href="https://pola-rs.github.io/nodejs-polars/index.html">Node.js</a> - <a href="https://rpolars.github.io/index.html">R</a> | <b>StackOverflow</b>: <a href="https://stackoverflow.com/questions/tagged/python-polars">Python</a> - <a href="https://stackoverflow.com/questions/tagged/rust-polars">Rust</a> - <a href="https://stackoverflow.com/questions/tagged/nodejs-polars">Node.js</a> - <a href="https://stackoverflow.com/questions/tagged/r-polars">R</a> | <a href="https://docs.pola.rs/">User guide</a> | <a href="https://discord.gg/4UfP5cfBE7">Discord</a> </p> ## Polars: Blazingly fast DataFrames in Rust, Python, Node.js, R, and SQL Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using [Apache Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html) as the memory model. - Lazy | eager execution - Multi-threaded - SIMD - Query optimization - Powerful expression API - Hybrid Streaming (larger-than-RAM datasets) - Rust | Python | NodeJS | R | ... To learn more, read the [user guide](https://docs.pola.rs/). ## Python ```python >>> import polars as pl >>> df = pl.DataFrame( ... { ... "A": [1, 2, 3, 4, 5], ... "fruits": ["banana", "banana", "apple", "apple", "banana"], ... "B": [5, 4, 3, 2, 1], ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"], ... } ... ) # embarrassingly parallel execution & very expressive query language >>> df.sort("fruits").select( ... "fruits", ... "cars", ... pl.lit("fruits").alias("literal_string_fruits"), ... pl.col("B").filter(pl.col("cars") == "beetle").sum(), ... pl.col("A").filter(pl.col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), ... pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"), ... pl.col("A").reverse().over("fruits").alias("rev_A_by_fruits"), ... pl.col("A").sort_by("B").over("fruits").alias("sort_A_by_B_by_fruits"), ... ) shape: (5, 8) ┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐ │ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │ │ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │ │ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │ │ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡ │ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │ │ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │ │ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │ │ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │ │ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │ └──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘ ``` ## SQL ```python >>> df = pl.scan_ipc("file.arrow") >>> # create a SQL context, registering the frame as a table >>> sql = pl.SQLContext(my_table=df) >>> # create a SQL query to execute >>> query = """ ... SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM my_table ... WHERE id1 = 'id016' ... LIMIT 10 ... """ >>> ## OPTION 1 >>> # run the query, materializing as a DataFrame >>> sql.execute(query, eager=True) shape: (1, 2) ┌────────┬────────┐ │ sum_v1 ┆ min_v2 │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════╪════════╡ │ 298268 ┆ 1 │ └────────┴────────┘ >>> ## OPTION 2 >>> # run the query but don't immediately materialize the result. >>> # this returns a LazyFrame that you can continue to operate on. >>> lf = sql.execute(query) >>> (lf.join(other_table) ... .group_by("foo") ... .agg( ... pl.col("sum_v1").count() ... ).collect()) ``` SQL commands can also be run directly from your terminal using the Polars CLI: ```bash # run an inline SQL query > polars -c "SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10" # run interactively > polars Polars CLI v0.3.0 Type .help for help. > SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10; ``` Refer to the [Polars CLI repository](https://github.com/pola-rs/polars-cli) for more information. ## Performance 🚀🚀 ### Blazingly fast Polars is very fast. In fact, it is one of the best performing solutions available. See the [TPC-H benchmarks](https://www.pola.rs/benchmarks.html) results. ### Lightweight Polars is also very lightweight. It comes with zero required dependencies, and this shows in the import times: - polars: 70ms - numpy: 104ms - pandas: 520ms ### Handles larger-than-RAM data If you have data that does not fit into memory, Polars' query engine is able to process your query (or parts of your query) in a streaming fashion. This drastically reduces memory requirements, so you might be able to process your 250GB dataset on your laptop. Collect with `collect(streaming=True)` to run the query streaming. (This might be a little slower, but it is still very fast!) ## Setup ### Python Install the latest Polars version with: ```sh pip install polars ``` We also have a conda package (`conda install -c conda-forge polars`), however pip is the preferred way to install Polars. Install Polars with all optional dependencies. ```sh pip install 'polars[all]' ``` You can also install a subset of all optional dependencies. ```sh pip install 'polars[numpy,pandas,pyarrow]' ``` See the [User Guide](https://docs.pola.rs/user-guide/installation/#feature-flags) for more details on optional dependencies To see the current Polars version and a