With Polars 2.0 defaulting to the streaming engine for massive performance gains, are you willing to trade guaranteed row order for speed?
Polars 2.0: Embracing the Streaming Engine and Stricter Data Types
Ritchie Vink has announced the first release candidate for Polars 2.0, marking a significant evolution for the high-performance data library. The most impactful change is that all LazyFrame queries will now default to the streaming engine. This shift is designed to provide massive performance benefits, with expected speed increases of up to 5x and much more efficient memory usage. However, this comes with a notable trade-off: the streaming engine does not guarantee row order for operations like joins and group_bys by default. To maintain order, users must now explicitly set `maintain_order=True`. Beyond performance, Polars 2.0 introduces stricter behavior to eliminate silent bugs, such as preventing lossy type conversions that previously occurred during `is_in` operations. These changes are particularly beneficial for the rise of AI-driven development, as stricter schema validation allows AI agents to catch errors early. While the update aims for a 'boring' transition, developers should prepare for these new defaults and the increased precision in data handling.
Options
- Prioritize performance: 5x speed is worth the trade-off.
- Prioritize stability: I need guaranteed row order.
- Support the change: Stricter error catching is vital.
- Avoid the shift: I'll stick to the in-memory engine.