🏆 Highlights

• Add support for IO[bytes] and bytes in scan_{...} functions (#18532)
• Add IEJoin algorithm for non-equi joins and support Full non-equi joins (#18365)

🚀 Performance improvements

• Back arrow arrays with SharedStorage which can have non-refcounted static slices (#18666)
• Don't traverse file list twice for extension validation (#18620)
• Remove cloning of ColumnChunkMetadata (#18615)
• Add upfront partitioning in ColumnChunkMetadata (#18584)
• Enable Parquet parallel=prefiltered for auto (#18514)
• Change PlSmallStr impl from Arc<str> to compact_str (#18508)
• Added optimizer rules for is_null().all() and similar expressions to use null_count() (#18359)
• Parquet do not copy uncompressed pages (#18441)
• Several large parquet optimizations (#18437)
• Batch Plain Parquet UTF-8 verification (#18397)
• Partition metadata for parquet statistic loading (#18343)
• Fix accidental quadratic parquet metadata (#18327)
• Lazy decompress Parquet pages (#18326)
• Don't rechunk aligned chunks in owned_binary_chunk_align (#18314)
• Batch DELTA_LENGTH_BYTE_ARRAY decoding (#18299)
• Slice pushdown for SimpleProjection (#18296)
• Use direct path for time/timedelta literals (#18223)
• Speedup ndjson reader ~40% (#18197)

✨ Enhancements

• Add support for IO[bytes] and bytes in scan_{...} functions (#18532)
• Add IEJoin algorithm for non-equi joins and support Full non-equi joins (#18365)
• Make expressions containing Python UDFs serializable (#18135)
• Support Serde for IRPlan (#18433)
• Respect input time zone if input is pandas Timestamp (#18346)
• Add POLARS_BACKTRACE_IN_ERR for debugging (#18333)
• IR serde (#18298)
• Improve decimal_comma error message (#18269)
• Support pre-signed URLs for cloud scan (#18274)
• Support empty structs (#18249)
• Allow float in interpolate_by by column (#18015)

🐞 Bug fixes

• Scalar checks (#18627)
• Scanning hive partitioned files where hive columns are partially included in the file (#18626)
• Enable "polars-json/timezones" feature from "polars-io" (#18635)
• Use Buffer<T> in ObjectSeries, fixes variety of offset bugs (#18637)
• Properly slice validity mask on pl.Object series (#18631)
• Indicative error in list.gather when wrong indices type is supplied (#18611)
• Fix group first value after group-by slice (#18603)
• Functions for streaming require streaming feature (#18602)
• Allow for date/datetime subclasses (e.g. pd.Timestamp, FreezeGun) in pl.lit (#18497)
• Fix UnitVec inline clone and with_capacity (#18586)
• Ensure result name of pow matches schema in grouped context (#18533)
• Decimal mean agg dtype was incorrect in IR (#18577)
• Fix output type for list.eval in certain cases (#18570)
• Fix map_elements for List return dtypes (#18567)
• Do not remove double-sort if maintain_order=True (#18561)
• Empty any_horizontal should be false, not true (#18545)
• Fix type inference error in map_elements for List types (#18542)
• Added proper handling of file.write for large remote csv files (#18424)
• Handle Parquet projection pushdown with only row index (#18520)
• Properly raise on invalid selector expressions (#18511)
• Wrong output column name in or and xor operations (#18512)
• Various schema corrections (#18474)
• Don't drop objects on empty buffers (#18469)
• Add missing chunk align in pipe sink (#18457)
• Expr.sign should preserve dtype (#18446)
• Enable CSE in eager if struct are expanded (#18426)
• Treat explode as gather (#18431)
• Fencepost error in debug assertion in splitfields (#18423)
• Unsoundness in CSV SplitFields (#18413)
• Parquet nested values that span several pages (#18407)
• Support reading empty parquet files (#18392)
• Recurse on map field during type conversion (#15075)
• Allow search_sorted on boolean series (#18387)
• Mark Expr.(lower|upper)_bound as returning scalar (#18383)
• Fix broken feature gate for ParquetReader (#18376)
• Fix compressed ndjson row count (#18371)
• Use correct column names when there are no value columns in unpivot (#18340)
• Parquet several smaller issues (#18325)
• Fix group-by slice on all keys (#18324)
• Compute joint null mask before calling rolling corr/cov stats (#18246)
• Several scan_parquet(parallel='prefiltered') problems (#18278)
• Json feature flag missing imports (#18305)
• Check groups in group-by filter (#18300)
• Make json readers ignore BOM character (#18240)
• Parquet delta encoding for 0-bitwidth miniblocks (#18289)
• Arguments for upsample only have to be sorted within groups (#18264)
• Use appropriate bins in hist when bin_count specified (#16942)
• Raise suitable error on unsupported SQL set op syntax (#18205)
• Fix invalid state due to cached IR (#18262)
• Fix failed AWS credential load from '~/.aws/credentials' due to formatting (#18259)
• Fix panic streaming parquet scan from cloud with slice (#18202)
• Consistently round half-way points down in dt.round (#18245)
• Fix duplicate column output and panic for include_file_paths (#18255)
• Fix unit null rank (#18252)
• Use physical for row-encoding (#18251)

📖 Documentation

• Fix multiprocessing docs regarding fork method check (#18563)
• Pre-compute plugin_path before defining plugin (#18503)
• Fix BinViewChunkedBuilder arguments (#17277) (#18439)
• Add date_range and datetime_ranges examples without eager=True (#18379)
• Document POLARS_BACKTRACE_IN_ERR env var (#18354)
• Document DataFrame.__getitem__ and Series.__getitem__ (#18309)
• Improve decimal_comma error message (#18269)
• Clarify coalesce behaviour in join_asof (#18273)
• Add note to Expr.shuffle differentiating from df method (#18266)

📦 Build system

• Remove extension-module from polars-python (#18554)
• Bump Rust toolchain to nightly-2024-08-26 (#18370)

🛠️ Other improvements

• Push down max row group height calc to file metadata (#18674)
• Re-use already decoded metadata for first path (new-parquet-source) (#18656)
• Remove duplicate byte range calc from new parquet source (#18655)
• Fix a bunch of tests for new-streaming (#18659)
• Rename MemSlice::from_slice -> MemSlice::from_static (#18657)
• Don't raise on multiple same names in ie_join (#18658)
• Split parquet_source.rs in new-streaming (#18649)
• Check predicates in join_where (#18648)
• Feature gate iejoin (#18646)
• Scan from BytesIO in new-streaming parquet source (#18643)
• Rename MetaData -> Metadata (#18644)
• Change join_where semantics (#18640)
• Fix unimplemented panics to give todo!s for AUTO_NEW_STREAMING (#18628)
• Remove extra schema traits (#18616)
• One simplify expression module and keep utility local (#18621)
• Check number of binary comparisons in join_where predicates (#18608)
• Raise on suffixed predicate in join_where (#18607)
• Fix Python docs build (#18605)
• Fix nan-ignoring max/min in new-streaming (#18593)
• Correctly support more types in new-streaming sum (#18580)
• Bump NodeTraverser major version (#18576)
• Fix mean reduction in new-streaming (#18572)
• Rename data_type -> dtype (#18566)
• Refactor ArrowSchema to use polars_schema::Schema<D> (#18564)
• Remove NotifyReceiver from new-streaming parquet source (#18540)
• Refactor Schema to use generic struct from new polars-schema crate (#18539)
• Temporarily pin NumPy in CI to address dependency resolving issue (#18544)
• Fix and extend AnyValue comparison (#18534)
• Remove top-level metadata from ArrowSchema (#18527)
• Add FromIterator impls for PlSmallStr (#18509)
• Update PlSmallStr comment (#18518)
• Change PlSmallStr impl from Arc<str> to compact_str (#18508)
• Make expressions containing Python UDFs serializable (#18135)
• Allow polars to pass cargo check on windows (#18498)
• Remove From<&&str> for PlSmallStr (#18507)
• Change naming to new benchmark setup (#18473)
• More refactor for PlSmallStr (#18456)
• Split Reduction into it plus ReductionState (#18460)
• Remove a string allocation in Parquet (#18466)
• Unify internal string type (#18425)
• Remove network call in hf docs (#18454)
• Remove old streaming flag if we're going into new streaming (#18438)
• Address spurious hypothesis test failure (#18434)
• Add pl.length() reduction and small new-streaming fixes (#18429)
• Fencepost error in debug assertion in splitfields (#18423)
• Group arguments in conversion in a Context (#18418)
• Turn all Binary/Utf8 into BinaryView/Utf8View in Parquet (#18331)
• Recursively evaluate is_elementwise for function expressions (#18385)
• Various small fixes for the new streaming engine (#18384)
• Temporarily add ability to disable parquet source node (#18378)
• Improve dot formatting of new-streaming parquet source (#18367)
• Fix the required version of rust in README.md (#18357)
• Only instantiate used portion of graph (#18337)
• Fix new_streaming parameter (#18342)
• Add parquet source node to new streaming engine (#18152)
• Disable common sub-expr elim for new streaming engine (#18330)
• Remove unused Parquet indexes (#18329)
• Lower arbitrary expressions in the new streaming engine (#18315)
• Expose many more function expressions to python IR (#18317)
• Add Graphviz physical plan visualization for new streaming engine (#18307)
• Add DataFrame::new_with_broadcast and simplify column uniqueness checks (#18285)
• Add output_schema to all PhysNodes (#18272)
• Change fn schema to fn collect_schema (#18261)
• Add multiplexer node to new streaming engine (#18241)
• Add feature gates for polars-python crate (#18232)
• Split py-polars crate (#18204)
• Update the required version of rust in README.md (#18203)
• Add itertools in utils (#18213)
• Use or_else for raising (#18206)
• Remove unused Parquet source files (#18193)

Thank you to all our contributors for making this release possible! @0xbe7a, @BartSchuurmans, @ChayimFriedman2, @MarcoGorelli, @StepfenShawn, @WbaN314, @adamreeve, @agossard, @alexander-beedie, @alonme, @barak1412, @cgbur, @coastalwhite, @corwinjoy, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @henryharbeck, @ion-elgreco, @jqnatividad, @krasnobaev, @liufeimath, @markxwang, @mcrumiller, @megaserg, @nameexhaustion, @orlp, @philss, @r-brink, @ritchie46, @skellys, @squnit, @stinodego, @sunadase, @thomascamminady and @wence-