What's Changed

🎉 Exciting New Features

🧵 [Mage IO] Weaviate Integration

🤔 Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4158

🔍 [Mage IO] Alogia Integration

Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4198

💥 Dynamic SQL Blocks

Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. 🎉

by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4430

🌊 [Kafka Streaming Sources] Offset & Partitions

For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning, End, Int, & Timestamp. This allows users to set specific positions inside a topic to consume data.

by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4226

⛴️ Configure Kubernetes Affinity & Tolerations

Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity and tolerations in your Kubernetes settings. ⚓

Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.

Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.

These configurations should help our Kubernetes power users make the most of Mage! 🧙

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4407

🐛 Bug Fixes

• Fix LDAP unpacking by @dy46 in https://github.com/mage-ai/mage-ai/pull/4399
• Fix reduce output and triple layout saving by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4409
• Fix file browser bug and Git push bug by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4411
• Fix serializing list and dict when formatting output by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4412
• Prevent index out of bounds by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4425
• Remove test print statements by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4431
• Removing Draft7 validation from Clickhouse destination by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4424
• Fix global_vars context in pipeline executor by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4435
• Fix a few global data product bugs by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4440
• Fix dynamic blocks OOM round 2 by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4445
• Fix incremental sync in chargebee source by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4450
• Don’t count values if is None by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4454
• Fix policy issue updating settings by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4456
• Fix keyboard shortcuts when its empty array by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4458
• Run submodule sync by @dy46 in https://github.com/mage-ai/mage-ai/pull/4457
• Fixing Snowflake write_pandas issue by @Luishfs in https://github.com/mage-ai/mage-ai/pull/4395
• Fix writing to Snowflake with mixed int and str types by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4460
• Fix some bugs and improve the edit page by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4462
• Fix error logging in pipeline executor by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4468
• When clicking show file versions in arcane library, show right panel by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4472
• Catch BigQuery if it fails to fix table names by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4479
• Fix pipeline detail prop passed for fetching files by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4485
• Always show edit pipeline button by @dy46 in https://github.com/mage-ai/mage-ai/pull/4484
• Revert change to PG IO by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4486
• Fix io redshift by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4487
• Fix multi project flag by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4490
• Fix Bigquery clean column name by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4500
• Convert datetime type for s3 data integration destination by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4501
• Fix dynamic child block outputs by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4422
• Fix database missing and serializing QueryJob by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4428
• Fix incorrect spelling by @ckfear in https://github.com/mage-ai/mage-ai/pull/4438
• Fix kafka type and doc by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4466

💅 Enhancements & Polish

• Workspace improvements by @dy46 in https://github.com/mage-ai/mage-ai/pull/4469
• Scheduler improvements by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4467
• Prevent unnecessary initial pipeline run by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4291
• Speed up monitor stats and reduce calls on overview by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4408
• Bump up dependency versions to resolve vulnerabilities by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4433
• Limit dynamic block output in notebook by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4436
• Improve bigquery name parsing to help fill in full name by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4447
• Add error logging by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4448
• Fix app slowness due to project platform check by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4483
• Add PVC retention policy by @dy46 in https://github.com/mage-ai/mage-ai/pull/4491
• E2e test for /pipelines by @edmondwinston in https://github.com/mage-ai/mage-ai/pull/4306
• Add query decorator for data integration blocks by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/4465

New Contributors

• @ckfear made their first contribution in https://github.com/mage-ai/mage-ai/pull/4438

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.60...0.9.62