What's Changed

🎉 Exciting New Features

Note: many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.

🎮 Multi-project Platform [BETA]

We've reworked our support for mulitple projects with a new multi-project platform! @tommydangerous is back at it again with this huge feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/80a88de4-615c-431a-9d8f-1d9e61af523a

If you'd like to try out the multi-project platform, you can check out this repo for a sample structure. Head over to your Mage settings to enable

• Support nested projects and custom code paths by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4161
• Trigger and run pipelines across projects by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4186
• Split pipeline scheduler and schedule models for project platform by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4233
• Configure root project preferences and settings by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4234
• Fix pipeline schedule creation and repo name by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4247
• Don't try to use file source if not exist by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4196
• Fix the way we store pipelines by type in the cache by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4278

🚀 Command Center [BETA]

Another new & exciting feature this week— the Mage Command Center. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/a01a7f11-a0c8-43f1-9a44-253202659c75

Enable the Command Center in settings and give it a spin today!

• Command center by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4249
• Add command center models by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4254

🪣 Bitbucket Version Control

Shout out to @dy46 for continuing to crush the version control integrations!

You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more here.

by @dy46 in https://github.com/mage-ai/mage-ai/pull/4176

◔ Qdrant integration

Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4081

🧱 dbt DX v2 + dbt Upgrade [BETA]

This release contains a huge dbt overhaul 🤯

Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. @tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!

Here's a quick demo:

https://github.com/mage-ai/mage-ai/assets/59450879/c6518d08-fd32-4bbb-b271-9aa28409b41d

• Upgrade dbt to 1.7 by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4244
• dbt v2 browser UI by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4200
• Add block browser modal to notebook by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4246
• Add dbt cache by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4193
• Add fields to dbt code block 2.0 for manual entry by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4331
• Use absolute paths in dbt block by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4307
• Add custom code block tags by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4250
• Collapse or expand folders by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4203

🐛 Bug Fixes

SQL blocks

• Remove double quotes for postgres by @dy46 in https://github.com/mage-ai/mage-ai/pull/4170
• Escape BigQuery project name in SQL block by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4294
• Pass in query_vars as a dict by @dy46 in https://github.com/mage-ai/mage-ai/pull/4280

Data integration

• Fix syncing MySQL TIME type by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4275
• Bump Google Ads version from 14 to 15 by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4289
• Fix incremental sync bug with missing arg by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4315
• Fix executing data integration block with ecs executor by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4322
• Wrap MSSQL table name with double quotes in data integration pipeline by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4290

Trigger and scheduling

• Use UTC date for trigger start date by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4283
• Display correct default start datetime when editing trigger/backfill by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4292
• Try preventing creating duplicate pipeline runs in scheduler by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4311
• Wrap block run initialization logic with lock by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4296

dbt

• Fix dbt profiles interpolation by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4225
• Lazy import for dbt files by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4251
• Re-work dbt project path for yaml files by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4207
• Error handling for "project not found" in dbt by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4334
• Adjust error behavior when adding dbt files by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4191

File browser

• Fix files page not opening files by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4222
• File browser bug bash by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4237
• Fix bug when deleting block from file browser by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4243
• Fix requestIdleCallback not supported on Safari by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4284
• Fix file browser not refreshing by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4223

Dynamic blocks

• Fix dynamic block + dynamic child blocks spawning other blocks by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4295
• Fix reduce output bug by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4326

Git

• Add actions to GitBranchPolicy by @dy46 in https://github.com/mage-ai/mage-ai/pull/4213
• Fix Git bugs by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4264
• Fix Git submodule sync by @dy46 in https://github.com/mage-ai/mage-ai/pull/4316
• Fix missing Git module by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4206

Other

• Fix silent errors from global hooks by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4173
• Fix block sorting bug by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4175
• Avoid saving error details in block run DB by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4179
• Reset page after applying pipeline filters by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4183
• Restrict opentelemetry package versions by @dy46 in https://github.com/mage-ai/mage-ai/pull/4208
• Fix dataframe validation by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4255
• Fix add new button tooltips by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4263
• Save statistics.json in correct execution partition folder by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4271
• Fix dashboard resizing by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4282
• Only set the schema in the DB when the server is started by @dy46 in https://github.com/mage-ai/mage-ai/pull/4293
• Fix several bugs on Pipelines dashboard by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4300
• Improve UI in several areas by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4309
• Fix dragging and dropping file in browser by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4312
• Fix interpolating mage secret in project metadata.yaml by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4319
• Remove undefined request by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4324
• Fix triple layout after width by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4325
• Fix cannot unpack non-iterable NoneType object by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4327
• Add small fixes by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4329
• Bug fixes and minor improvements on version control app by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4267
• Fix a ton of bugs by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4214
• Fix escaping quotes in R block by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4285

💅 Enhancements & Polish

File browser

• Improve folder performance by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4221
• View or hide hidden files in file browser by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4235
• Re-use file browser and editor in notebook by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4231

Raw SQL enhancements

• Disable sample query when using raw SQL by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4252
• Add execute raw SQL without preprocessing to all loaders by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4303

Workspace management

• Add default values for workspace creation by @dy46 in https://github.com/mage-ai/mage-ai/pull/4219
• Use current pod image as the default by @dy46 in https://github.com/mage-ai/mage-ai/pull/4299
• Update url logic for workspace manager

Other

• Update Git settings when environment variables are set in https://github.com/mage-ai/mage-ai/pull/4154 in https://github.com/mage-ai/mage-ai/pull/4166
• Global hooks for platform by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4236
• Decrease image size by removing Qdrant libraries by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4277
• Update Git to set username and email when using version control by @dy46 in https://github.com/mage-ai/mage-ai/pull/4262
• Dynamic dynamic child and replicas by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4308
• Speed up pipeline run count query by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4321
• Rewrite schedule queries by @dy46 in https://github.com/mage-ai/mage-ai/pull/4185
• Update custom block template by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4177
• Support serialize Polars dataframe by @dy46 in https://github.com/mage-ai/mage-ai/pull/4281
• Add queue_subscription support for NATS Jetstream streaming source by @atjhoendz in https://github.com/mage-ai/mage-ai/pull/4169
• Added URL protocol option to RabbitMQ streaming by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4228
• AWS S3 Destination: Added support for IAM role-based authentication by @eofs in https://github.com/mage-ai/mage-ai/pull/4320
• Added role_arn as an optional config for AWS DynamoDB source by @robreinhold in https://github.com/mage-ai/mage-ai/pull/4256
• SQL block 1.5 by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4302
• Add retry info in callbacks and retry function by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4332
• Add custom 404 page by @edmondwinston in https://github.com/mage-ai/mage-ai/pull/4149

New Contributors

• @aliavni made their first contribution in https://github.com/mage-ai/mage-ai/pull/4181
• @atjhoendz made their first contribution in https://github.com/mage-ai/mage-ai/pull/4169
• @robreinhold made their first contribution in https://github.com/mage-ai/mage-ai/pull/4256

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.50...0.9.59