Polars

@pola.rs

Dataframes powered by a multithreaded, vectorized query engine, written in Rust.

We've released Polars Cloud client 0.10.0: • Stream query results into Python with `sink_batches()` • A new experimental query planner: Miso • Distributed `pl.collect_all()` • Hive-partition aware scans • On-Prem HDFS support Blog post: pola.rs/posts/polars... Changelog: github.com/pola-rs/pola...

Announcing Polars Cloud 0.10.0

Polars Cloud 0.10.0 streams distributed query results into Python with sink_batches(), adds an experimental miso query planner, distributed pl.collect_all(), experimental HDFS support, optimized hive-...

pola.rs

A float column can hold two different kinds of missing: null for a value that is absent, and NaN for arithmetic that had no valid answer (think 0.0/0.0). Polars keeps them strictly separate.

Bild

Reading a terabyte dataset from S3 goes fastest on a cluster of small machines, while heavy joins run fastest on one big machine. We benchmarked single node Polars against distributed Polars on the same total resources, and the bottleneck of your query decides the winner. pola.rs/posts/single...

Bild

We've released Polars Cloud client 0.9.0. • Expressions now run distributed • 17% Faster cloud I/O • Breaking: `ClusterContext` now uses `uri=` The `compute_address=` keyword is superseded by `uri=`. Blog post: pola.rs/posts/polars... Full changelog: github.com/polars-inc/p...

Announcing Polars Cloud 0.9.0

Polars Cloud 0.9.0 runs expressions distributed, ships large cloud I/O performance gains, a reworked ClusterContext API, a distributed Iceberg sink, and more.

pola.rs

How well do LLMs migrate pandas to Polars by themselves? We tested how well Claude translates a pandas corpus to Polars. Results were promising but not perfect. To improve this, we built a Polars skill that helps the agent. Read the full post here: pola.rs/posts/llm-po...

Bild

Distributed Polars is 3x faster than Spark on the PDSH benchmark and up to 7.8x faster on individual queries. Read the full benchmark post here: https:/pola.rs/posts/polars-pyspark-benchmarks/

Figure showing the speed-up of Polars compared to PySpark per query of the PDS-H benchmark.

Run Polars' distributed engine on your own infrastructure. Deploy a distributed Polars cluster on any Kubernetes setup (EKS, AKS, GKE, or minikube) and get a query dashboard with past queries, advanced query profiling, Open-lineage support, and more. More at pola.rs/posts/polars...

Bild

Polars supports a full Iceberg roundtrip on the streaming engine. You can scan an Iceberg table with scan_iceberg(), transform it lazily, and write the result back with sink_iceberg(). Useful for workflows like data redaction or compliance cleanup.

Bild

We've released Python Polars 1.40. The following now runs on the streaming engine • AsOf joins with a `by` argument • Basic elementwise over() • More expressions: cov(), corr(), interpolate(), skew(), kurtosis(), and entropy() and much more! github.com/pola-rs/pola...

Release Python Polars 1.40.0 · pola-rs/polars

🏆 Highlights Add streaming support for grouped AsOf join (#27293) ⚠️ Deprecations Deprecate support for dataframe interchange protocol (#27214) 🚀 Performance improvements Create IR slice from ...

github.com

Realtime query profiling of Polars In this post we use the query profiler in Polars Cloud to optimize the infrastructure configuration for a specific query. This results in a 54% faster and 64% cheaper query with only five runs. Read all about it here: pola.rs/posts/query-...

We've released Polars Cloud client 0.6.0. Some of the highlights: • Improved UX for query profiling • Compute Scratchpad Alpha • Improved distributed query planning • Breaking: `LazyFrameRemote.execute` is now blocking by default

Bild

We've released Python Polars 1.39. Some of the highlights: • Streaming AsOf join, enabling memory-efficient time-series joins. • sink_iceberg() for writing to Iceberg tables • Streaming cloud downloads for scan_csv(), scan_ndjson(), and scan_lines() github.com/pola-rs/pola...

Release Python Polars 1.39.0 · pola-rs/polars

🚀 Performance improvements Lower arg_{min,max} to streaming engine (#26845) Additional IR slice pushdown after filter pushdown (#26815) Streaming first/last on Enum through physical (#26783) Fast ...

github.com

pl.from_repr() constructs a DataFrame or Series directly from its printed string representation. This can be useful in unit tests: instead of rebuilding expected DataFrames through dictionaries with typecasting, the schema is encoded in the header and the values are right there in the table.

Bild

str.len_bytes() vs str.len_chars() len_bytes: ~20x faster, counts UTF-8 bytes len_chars: counts actual Unicode characters - Use len_bytes for ASCII data (IDs, hashes) - Use len_chars for anything multilingual len_bytes is O(1) metadata lookup, len_chars is O(n) traversal.

Bild

We've released Python Polars 1.38. Some of the highlights: • (De)Compression support on text based sources and sinks • scan_lines() to read text files • Merge join in the Streaming engine Link to the complete changelog: github.com/pola-rs/pola...

Release Python Polars 1.38.0 · pola-rs/polars

⚠️ Deprecations Deprecate retries=n in favor of storage_options={"max_retries": n} (#26155) 🚀 Performance improvements Enable zero-copy object_store put upload for IPC sink (#26288) Resolve file...

github.com

We refactored the Categorical in 1.31. The new Categories object gives you: • Control over the physical type (UInt8/16/32) • Named categories with namespaces • Parallel updates without locks • Automatic garbage collection Full read: pola.rs/posts/catego...

In 1-2 weeks we land live query profiling in Polars Cloud. See exactly how many rows are consumed and produced per operation. Which operation takes most runtime, and watch the data flow through live, like water. 😍

Bild

We just released Polars 1.37, here are the highlights: Improved Streaming Sinks: 1.14x-1.88x speedup, ~10% of the original memory. Streaming Compressed CSVs Faster SQL Ordering pl.PartitionBy min_by / max_by (see below) Series.sql() Free-Threading Support Python 3.9 Support Dropped musl Builds

Bild

Did you know about pl.corr()? The problem with data aggregation is that it can hide what's really going on. Below you can find Simpson's Paradox Sometimes the devil really is in the details.

Bild