This one's a big one!

In brief, to get you to the goods quicker:

• New VARCHAR data type over STRING default for ++compression & ++performance
• 50% improvement over already fast SQL query performance
• Data compression via system-level ZFS, previously Enterprise-only

And more...

Upgrade as soon as you can. Or start fresh and build out your use case.

If you're still reading, onwards for more detail, breaking changes, plus the PR breakdown.

New features 🐣

• VARCHAR: Introduces the VARCHAR data type and a new type driver API. VARCHAR offers UTF8 encoded strings of variable length as opposed to the UTF16 and fixed length encoding found in STRING, for better compression and significantly improved performance.
  • Maximum size of a single VARCHAR field size is 268 MB.
  • A size of a VARCHAR column in a single partition is limited by 218 TB or disk size, whichever is smaller.
  • STRING is now considered a legacy type.
• Data compression: Data compression for OSS! Full support for the ZFS file system and system-level data compression.
  • Read our guide to setup ZFS.
• ALTER COLUMN type: Use ALTER TABLE SQL to convert column types.
  • Migrate between STRING, SYMBOL, and VARCHAR data types.

Breaking changes 💥

• GLIBC upgrade: Upgraded GLIBC to a minimum of 2.28-amd64 and 2.33-aarch64.
• ILP and TextImport changes: ILP and TextImport now create VARCHAR instead of STRING columns.
• Jemalloc allocator: The default allocator is now jemalloc. Later updates include disabling jemalloc as the default allocator and removing jemalloc from no-jre builds.

Performance 🚀

• SQL performance improvement: Average SQL performance improves by 50%. 🔥
• Memory allocation limit: New memory allocation limit prevents OS OOM handler terminations in many cases.

Bug fixes 🥾

• SQL: Fixed issues with SAMPLE BY and GROUP BY not grouping when used with column aliases. Fixed WHERE filters with FLOAT columns null comparison.
• Core: Addressed trailing boundary processing of multipart HTTP requests. Fixed hour parsing and formatting in the 1-12 and 1-24 conventions.

Pull requests :octocat:

• perf(sql): improve performance of length(VARCHAR) by @mtopolnik in https://github.com/questdb/questdb/pull/4442
• perf(sql): reduce query latency when querying tables under heavy writing by @ideoma in https://github.com/questdb/questdb/pull/4500
• perf(sql): statistics-based map pre-sizing for parallel group by by @puzpuzpuz in https://github.com/questdb/questdb/pull/4517
• perf(sql): speed up GROUP BY queries with ORDER BY and LIMIT by @puzpuzpuz in https://github.com/questdb/questdb/pull/4523
• perf(sql): optimize ORDER BY with LIMIT by @jerrinot in https://github.com/questdb/questdb/pull/4529
• perf(sql): adaptive work stealing for parallel query execution on multicore machines by @puzpuzpuz in https://github.com/questdb/questdb/pull/4542
• feat(core): breaking change💥 - upgrade GLIBC to minimum of 2.28-amd64 and 2.33-aarch64 by @bluestreak01 in https://github.com/questdb/questdb/pull/4449
• feat(core): use jemalloc allocator by default by @bluestreak01 in https://github.com/questdb/questdb/pull/4451
• feat(sql): implement level 2 order book price function by @nwoolmer in https://github.com/questdb/questdb/pull/4462
• feat(core): add ZFS support by @puzpuzpuz in https://github.com/questdb/questdb/pull/4507
• feat(sql): support usage of ALTER table SQL to convert column types by @ideoma in https://github.com/questdb/questdb/pull/4413
• feat(ilp): breaking change💥 - ILP and TextImport create VARCHAR instead of STRING columns by @jerrinot in https://github.com/questdb/questdb/pull/4476
• feat(build): new ami based on amazon linux 2023 💥 - breaking change by @mariusgheorghies in https://github.com/questdb/questdb/pull/4519
• feat(core): resume all suspended WAL tables on start up by @ideoma in https://github.com/questdb/questdb/pull/4513
• feat(core): add RSS memory limit to prevent OOM kills from the OS by @mtopolnik in https://github.com/questdb/questdb/pull/4480
• feat(sql): introduce global select query cache per interface by @puzpuzpuz in https://github.com/questdb/questdb/pull/4525
• feat(core): improve SQL error logging by @bluestreak01 in https://github.com/questdb/questdb/pull/4532
• fix(sql): fix sample by query compilation error when using as-of join by @amunra in https://github.com/questdb/questdb/pull/4452
• fix(core): fix trailing boundary processing of multipart HTTP requests by @glasstiger in https://github.com/questdb/questdb/pull/4466
• fix(http): remove ddl:ok response for insert query by @siddharth0815 in https://github.com/questdb/questdb/pull/4472
• fix(ui): web console to load regardless of pgwire disabled by @glasstiger in https://github.com/questdb/questdb/pull/4484
• fix(sql): fix WHERE filters with FLOAT columns null comparison by @ideoma in https://github.com/questdb/questdb/pull/4487
• fix(sql): minor bug fixes in ExpressionParser by @sivukhin in https://github.com/questdb/questdb/pull/4441
• fix(sql): IN with NULLs and STRINGs sometimes may return the wrong result by @nwoolmer in https://github.com/questdb/questdb/pull/4499
• fix(sql): fix SAMPLE BY and GROUP BY not grouping when used with column aliases by @ideoma in https://github.com/questdb/questdb/pull/4503
• fix(core): fix hour parsing and formatting in the 1-12 and 1-24 conventions by @jerrinot in https://github.com/questdb/questdb/pull/4506
• fix(sql): fix several bugs in aggregation (avg/sum/first_value) window functions by @sivukhin in https://github.com/questdb/questdb/pull/4429
• fix(sql): split_part returning old data when no match is found by @nwoolmer in https://github.com/questdb/questdb/pull/4493
• fix(sql): breaking💥 fix coalesce function correctness by @bluestreak01 in https://github.com/questdb/questdb/pull/4490
• fix(core): fix potential data loss when non-WAL table dropped and created with same name by @ideoma in https://github.com/questdb/questdb/pull/4533
• fix(sql): breaking change💥 - fix expression parser operator precedence by @sivukhin in https://github.com/questdb/questdb/pull/4443
• fix(core): fix transient query failure because of missing partition directory by @ideoma in https://github.com/questdb/questdb/pull/4611

New Contributors

• @siddharth0815 made their first contribution in https://github.com/questdb/questdb/pull/4472

Full Changelog: https://github.com/questdb/questdb/compare/7.4.2...8.0.0