What's Changed

• Adopt list_streaming in tenant deletion by @arpad-m in https://github.com/neondatabase/neon/pull/8504
• Add a test for clickhouse as a logical replication consumer by @a-masterov in https://github.com/neondatabase/neon/pull/8408
• test: deflake test_duplicate_creation by @koivunej in https://github.com/neondatabase/neon/pull/8536
• pageserver: fix return code from secondary_download_handler by @jcsp in https://github.com/neondatabase/neon/pull/8508
• pageserver: fail if id is present in pageserver.toml by @problame in https://github.com/neondatabase/neon/pull/8489
• l0_flush: use mode=direct by default => coverage in automated tests by @problame in https://github.com/neondatabase/neon/pull/8534
• scrubber: enable cleaning up garbage tenants from known deletion bugs, add object age safety check by @jcsp in https://github.com/neondatabase/neon/pull/8461
• Add metrics for input data considered and taken for compression by @arpad-m in https://github.com/neondatabase/neon/pull/8522
• scrubber: add remote_storage based listing APIs and use them in find-large-objects by @arpad-m in https://github.com/neondatabase/neon/pull/8541
• CI(benchmarking): make neonvm default provisioner by @bayandin in https://github.com/neondatabase/neon/pull/8538
• [neon/acr] impr: push to ACR while building images by @antonyc in https://github.com/neondatabase/neon/pull/8545
• feat(storcon): store scrubber metadata scan result by @yliang412 in https://github.com/neondatabase/neon/pull/8480
• feat(scrubber): post scan_metadata results to storage controller by @yliang412 in https://github.com/neondatabase/neon/pull/8502
• cleanup(compact_level0_phase1): some commentary and wrapping into block expressions by @problame in https://github.com/neondatabase/neon/pull/8544
• cicd: change Azure storage details [1/2] by @fcdm in https://github.com/neondatabase/neon/pull/8553
• compaction_level0_phase1: bypass PS PageCache for data blocks by @problame in https://github.com/neondatabase/neon/pull/8543
• CI(regress-tests): run less regression tests by @bayandin in https://github.com/neondatabase/neon/pull/8561
• feat(pageserver): support btm-gc-compaction for child branches by @skyzh in https://github.com/neondatabase/neon/pull/8519
• refactor(page_service): Timeline gate guard holding + cancellation + shutdown by @problame in https://github.com/neondatabase/neon/pull/8339
• Fix negative replication delay metric by @tristan957 in https://github.com/neondatabase/neon/pull/8520
• cicd: change Azure storage details [2/2] by @fcdm in https://github.com/neondatabase/neon/pull/8562
• controller: simplify reconciler generation increment logic by @jcsp in https://github.com/neondatabase/neon/pull/8560
• storage_scrubber: migrate FindGarbage to remote_storage by @arpad-m in https://github.com/neondatabase/neon/pull/8548
• CI(create-test-report): fix missing benchmark results in Allure report by @bayandin in https://github.com/neondatabase/neon/pull/8540
• test(pageserver): add test_gc_feedback_with_snapshots by @skyzh in https://github.com/neondatabase/neon/pull/8474
• pageserver: shutdown all walredo managers 8s into shutdown by @problame in https://github.com/neondatabase/neon/pull/8572
• Reduce linux-raw-sys duplication by @arpad-m in https://github.com/neondatabase/neon/pull/8577
• pageserver: add layer visibility calculation by @jcsp in https://github.com/neondatabase/neon/pull/8511
• storage-scrubber: log version on start by @skyzh in https://github.com/neondatabase/neon/pull/8571
• fix(pageserver): skip existing layers for btm-gc-compaction by @skyzh in https://github.com/neondatabase/neon/pull/8498
• test_runner: don't create artifacts if Allure is not enabled by @bayandin in https://github.com/neondatabase/neon/pull/8580
• pageserver: refine how we delete timelines after shard split by @jcsp in https://github.com/neondatabase/neon/pull/8436
• storage_controller: start adding chaos hooks by @jcsp in https://github.com/neondatabase/neon/pull/7946
• Wait for completion of the upload queue in flush_frozen_layer by @arpad-m in https://github.com/neondatabase/neon/pull/8550
• Improve safekeepers eviction rate limiting by @petuhovskiy in https://github.com/neondatabase/neon/pull/8456
• tests: add test_historic_storage_formats by @jcsp in https://github.com/neondatabase/neon/pull/8423
• fix(pageserver): deadlock in gc-compaction by @skyzh in https://github.com/neondatabase/neon/pull/8590
• feat(pageserver): support auto split layers based on size by @skyzh in https://github.com/neondatabase/neon/pull/8574
• safekeeper: remove unused partial_backup_enabled option by @dotdister in https://github.com/neondatabase/neon/pull/8547
• feat(pageserver): support split delta layers by @skyzh in https://github.com/neondatabase/neon/pull/8599
• Update Postgres versions to use smgrexists() instead of access() to check if Oid is used by @knizhnik in https://github.com/neondatabase/neon/pull/8597
• CI(trigger-e2e-tests): wait for promote-images job from the last commit by @bayandin in https://github.com/neondatabase/neon/pull/8592
• CI(trigger-e2e-tests): fix deadlock with Build and Test workflow by @bayandin in https://github.com/neondatabase/neon/pull/8606
• fix: allow awaiting logical size for root timelines by @koivunej in https://github.com/neondatabase/neon/pull/8604
• CI(benchmarking): set pub/sub projects for LR tests by @bayandin in https://github.com/neondatabase/neon/pull/8483
• feat(pageserver): support dry-run for gc-compaction, add statistics by @skyzh in https://github.com/neondatabase/neon/pull/8557
• fix: make Timeline::set_disk_consistent_lsn use fetch_max by @koivunej in https://github.com/neondatabase/neon/pull/8311
• feat: persistent gc blocking by @koivunej in https://github.com/neondatabase/neon/pull/8600
• pageserver: remove legacy read path by @VladLazar in https://github.com/neondatabase/neon/pull/8601
• chore: bump index part version by @koivunej in https://github.com/neondatabase/neon/pull/8611
• storage_scrubber: migrate scan_safekeeper_metadata to remote_storage by @arpad-m in https://github.com/neondatabase/neon/pull/8595
• Add package-mode=false to poetry. by @arssher in https://github.com/neondatabase/neon/pull/8609
• Add a test using Debezium as a client for the logical replication by @a-masterov in https://github.com/neondatabase/neon/pull/8568
• enable rum test by @a-masterov in https://github.com/neondatabase/neon/pull/8380
• tests: improve stability of test_storage_controller_many_tenants by @jcsp in https://github.com/neondatabase/neon/pull/8607
• pageserver: fixes for layer visibility metric by @jcsp in https://github.com/neondatabase/neon/pull/8603
• pageserver: use layer visibility when composing heatmap by @jcsp in https://github.com/neondatabase/neon/pull/8616
• pageserver: add bench_ingest by @jcsp in https://github.com/neondatabase/neon/pull/7409
• scrubber: clean up scan_metadata before prod by @yliang412 in https://github.com/neondatabase/neon/pull/8565
• storage broker: only print one line for version and build tag in init by @arpad-m in https://github.com/neondatabase/neon/pull/8624
• Lower level for timeline cancellations during gc by @arpad-m in https://github.com/neondatabase/neon/pull/8626
• Add missing colon to ArchivalConfigRequest specification by @arpad-m in https://github.com/neondatabase/neon/pull/8627
• proxy: random changes by @conradludgate in https://github.com/neondatabase/neon/pull/8602
• fix: EphemeralFiles can outlive their Timeline via enum LayerManager by @koivunej in https://github.com/neondatabase/neon/pull/8229
• fix(pageserver): dump the key when it's invalid by @skyzh in https://github.com/neondatabase/neon/pull/8633
• refactor(timeline_detach_ancestor): replace ordered reparented with a hashset by @koivunej in https://github.com/neondatabase/neon/pull/8629
• cicd: push build-tools image to ACR as well by @fcdm in https://github.com/neondatabase/neon/pull/8638
• fix: drain completed page_service connections by @koivunej in https://github.com/neondatabase/neon/pull/8632
• feat(pageserver): add direct io pageserver config by @yliang412 in https://github.com/neondatabase/neon/pull/8622
• pageserver: make bench_ingest build (but panic) on macOS by @jcsp in https://github.com/neondatabase/neon/pull/8641
• Use sycnhronous commit for logical replicaiton worker by @knizhnik in https://github.com/neondatabase/neon/pull/8645
• fix(test): do not fail test for filesystem race by @koivunej in https://github.com/neondatabase/neon/pull/8643
• fix: stop leaking BackgroundPurges by @koivunej in https://github.com/neondatabase/neon/pull/8650
• pageserver: remove paranoia double-calculation of retain_lsns by @jcsp in https://github.com/neondatabase/neon/pull/8617
• tests: don't require kafka client for regular tests by @jcsp in https://github.com/neondatabase/neon/pull/8662
• CI(build-tools): update Rust, Python, Mold by @bayandin in https://github.com/neondatabase/neon/pull/8667
• Dockerfiles: fix LegacyKeyValueFormat & JSONArgsRecommended by @bayandin in https://github.com/neondatabase/neon/pull/8664
• Run a subset of benchmarking job steps on GitHub action runners in Azure - closer to the system under test by @Bodobolero in https://github.com/neondatabase/neon/pull/8651
• Revert "proxy: update tokio-postgres to allow arbitrary config params (#8076)" by @conradludgate in https://github.com/neondatabase/neon/pull/8654
• fix(neon): disable create tablespace stmt by @skyzh in https://github.com/neondatabase/neon/pull/8657
• CI(pin-build-tools-image): fix permissions for Azure login by @bayandin in https://github.com/neondatabase/neon/pull/8671
• pageserver: don't treat NotInitialized::Stopped as unexpected by @jcsp in https://github.com/neondatabase/neon/pull/8675
• storcon: skip draining shard if it's secondary is lagging too much by @VladLazar in https://github.com/neondatabase/neon/pull/8644
• Dockerfiles: remove cachepot by @bayandin in https://github.com/neondatabase/neon/pull/8666
• Update docs/SUMMARY.md by @jcsp in https://github.com/neondatabase/neon/pull/8665
• Also pass HOME env var in access_env_vars by @arpad-m in https://github.com/neondatabase/neon/pull/8685
• chore(deps): bump aiohttp from 3.9.4 to 3.10.2 by @dependabot in https://github.com/neondatabase/neon/pull/8684
• CI: add actions/set-docker-config-dir to set DOCKER_CONFIG by @bayandin in https://github.com/neondatabase/neon/pull/8676
• Fix logical apply worker reporting of flush_lsn wrt sync replication. by @arssher in https://github.com/neondatabase/neon/pull/8669
• safekeeper membership change rfc by @arssher in https://github.com/neondatabase/neon/pull/8455
• Fix a comment in walproposer_pg.c by @shinyaaa in https://github.com/neondatabase/neon/pull/8583
• tests: reinstate test_bulk_insert by @jcsp in https://github.com/neondatabase/neon/pull/8683
• test: do graceful shutdown by default by @koivunej in https://github.com/neondatabase/neon/pull/8655
• storcon: implement graceful leadership transfer by @VladLazar in https://github.com/neondatabase/neon/pull/8588
• fix(walredo): shutdown can complete too early by @problame in https://github.com/neondatabase/neon/pull/8701
• Automatically prepare/restore Aurora and RDS databases from pg_dump in benchmarking workflow by @Bodobolero in https://github.com/neondatabase/neon/pull/8682
• Add on-demand WAL download to slot funcs by @save-buffer in https://github.com/neondatabase/neon/pull/8705
• Allow logical_replication_max_snap_files = -1 by @arssher in https://github.com/neondatabase/neon/pull/8702
• pageserver: add CompactKey, use it in InMemoryLayer by @jcsp in https://github.com/neondatabase/neon/pull/8652
• pageserver: remove vectored get related configs by @VladLazar in https://github.com/neondatabase/neon/pull/8695
• Report search_path to make it possible to use it in pgbouncer track_extra_parameters by @knizhnik in https://github.com/neondatabase/neon/pull/8303
• pageserver: implement utilization score by @jcsp in https://github.com/neondatabase/neon/pull/8703
• CI: mark PRs and issues create by external users by @bayandin in https://github.com/neondatabase/neon/pull/8694
• bench: stop immediatedly in some benches by @koivunej in https://github.com/neondatabase/neon/pull/8713
• fix: make compaction more sensitive to cancellation by @koivunej in https://github.com/neondatabase/neon/pull/8706
• refactor(test): unify how we clear shared buffers by @koivunej in https://github.com/neondatabase/neon/pull/8634
• Benchmarking tests: automatically restore Neon reuse databases, too and migrate to pg16 by @Bodobolero in https://github.com/neondatabase/neon/pull/8707
• test: do better job of shutting everything down by @koivunej in https://github.com/neondatabase/neon/pull/8714
• feat(detach_ancestor): better retries with persistent gc blocking by @koivunej in https://github.com/neondatabase/neon/pull/8430
• Decode the database name in SQL/HTTP connections by @tristan957 in https://github.com/neondatabase/neon/pull/8718
• Preserve HEAP_COMBOCID when restoring t_cid from WAL by @knizhnik in https://github.com/neondatabase/neon/pull/8503
• scrubber: fix spurious "Missed some shards" errors by @jcsp in https://github.com/neondatabase/neon/pull/8661
• timeline_detach_ancestor: adjust error handling by @koivunej in https://github.com/neondatabase/neon/pull/8528
• pageserver: evict covered layers earlier by @jcsp in https://github.com/neondatabase/neon/pull/8679
• CI(label-for-external-users): check membership using GitHub API by @bayandin in https://github.com/neondatabase/neon/pull/8724
• refactor: error/anyhow::Error wrapping by @koivunej in https://github.com/neondatabase/neon/pull/8697
• proxy: start of jwk cache by @conradludgate in https://github.com/neondatabase/neon/pull/8690
• storcon_cli: don't clobber heatmap interval when setting eviction by @jcsp in https://github.com/neondatabase/neon/pull/8722
• CI(build-neon): fix accidental neon rebuild on cargo test by @bayandin in https://github.com/neondatabase/neon/pull/8721
• chore: even more responsive compaction cancellation by @koivunej in https://github.com/neondatabase/neon/pull/8725
• CI(neon-image): add ARM-specific RUSTFLAGS by @bayandin in https://github.com/neondatabase/neon/pull/8566
• CI: misc cleanup & fixes by @bayandin in https://github.com/neondatabase/neon/pull/8559
• refactor(write path): newtype to enforce use of fully initialized slices by @problame in https://github.com/neondatabase/neon/pull/8717
• safekeeper: consider partial uploads when pulling timeline by @VladLazar in https://github.com/neondatabase/neon/pull/8628
• fix(pageserver): allow unused_imports in download.rs on macOS by @bayandin in https://github.com/neondatabase/neon/pull/8733
• storcon: deny external node configuration if an operation is ongoing by @koivunej in https://github.com/neondatabase/neon/pull/8727
• storcon: add spans to drain/fill ops by @koivunej in https://github.com/neondatabase/neon/pull/8735
• storcon: use tracing for logging panics by @koivunej in https://github.com/neondatabase/neon/pull/8734
• Handle reload config file request in LR monitor by @knizhnik in https://github.com/neondatabase/neon/pull/8732
• CI(label-for-external-users): use CI_ACCESS_TOKEN by @bayandin in https://github.com/neondatabase/neon/pull/8738
• CI(replication-tests): store test results & change notification channel by @bayandin in https://github.com/neondatabase/neon/pull/8687
• Add logical replication test to exercise snapfiles by @save-buffer in https://github.com/neondatabase/neon/pull/8364
• chore: log if we have an open layer or any frozen on shutdown by @koivunej in https://github.com/neondatabase/neon/pull/8740
• Add retroactive RFC about physical replication by @hlinnaka in https://github.com/neondatabase/neon/pull/8546
• tests: add infra and test for storcon leadership transfer by @VladLazar in https://github.com/neondatabase/neon/pull/8587
• safekeeper: cross check divergence point in ProposerElected handling. by @arssher in https://github.com/neondatabase/neon/pull/8612
• pageserver: prioritize secondary downloads to get most recent layers first, except l0s by @jcsp in https://github.com/neondatabase/neon/pull/8729
• Fix superuser check in test_snap_files by @save-buffer in https://github.com/neondatabase/neon/pull/8749
• Track holes to be able to reuse them once LFC limit is increased by @knizhnik in https://github.com/neondatabase/neon/pull/8575
• refactor(scrubber): add unified command suitable for cron job by @yliang412 in https://github.com/neondatabase/neon/pull/8635
• Default image compression to zstd at level 1 by @arpad-m in https://github.com/neondatabase/neon/pull/8677
• Merge main into release with merge commit by @arssher in https://github.com/neondatabase/neon/pull/8765
• Storage & Compute release 2024-08-19 by @vipvap in https://github.com/neondatabase/neon/pull/8751

Full Changelog: https://github.com/neondatabase/neon/compare/release-6299...release-6450