What's Changed

🎉 Exciting New Features

⚡️ Spark UI/UX 2.0 for AWS EMR

It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a huge update that comes with a complete overhaul of every element possible for managing your Spark cluster!

Check out the PR for more screenshots and get started today (docs coming soon)!

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/3997

🌊 Streaming: ActiveMQ Source

🔥 @shrutimantri is on fire! In another community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!

by @shrutimantri in https://github.com/mage-ai/mage-ai/pull/3978

📁 Download Files & Pipelines via the UI

Another community PR, this one from @PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.

This will be a game changer for collabrative projects and importing/exporting data from Mage!

by @PopaRares in https://github.com/mage-ai/mage-ai/pull/3813

✨ Streaming: NATS JetStream Source

A big shoutout to community member @mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!

by @mfreeman451 in https://github.com/mage-ai/mage-ai/pull/3985

🎏 Data integration: Kafka destination

@Luishfs is back at it with another destination— this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3976

🛟 Auto-save triggers in code

Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4009

🐛 Bug Fixes

• Add pyarrow-hotfix to requirements.txt by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3993
• Fix workspace user management fetching by @dy46 in https://github.com/mage-ai/mage-ai/pull/3992
• Handle hidden block positions when split view by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4005
• Don’t show SSH tunnel option unless kernel is PySpark by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4006
• Fix Monoco Editor for base path by @dy46 in https://github.com/mage-ai/mage-ai/pull/4012
• Update OAuth sign on and fix OAuth sign on with REQUIRE_USER_PERMISSIONS by @dy46 in https://github.com/mage-ai/mage-ai/pull/4007
• Clarification when applying bookmark to all streams by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4036 and https://github.com/mage-ai/mage-ai/pull/4049
• Allow using block configuration when run_pipeline_in_one_process is true by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4046
• Only show Backfills in vertical nav for standard (python) pipelines by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4050
• Fix files not being selected in notebook by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4051

💅 Enhancements & Polish

• Added application/gzip support to API source by @Luishfs in https://github.com/mage-ai/mage-ai/pull/3990
• Re-enqueue the job if queue is empty by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3996
• Add better error message for API source by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4016
• Pass envFrom to job pod by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4031

New Contributors

• @mfreeman451 made their first contribution in https://github.com/mage-ai/mage-ai/pull/3985