What's Changed

🎉 Exciting New Features

☁️ Google cloud storage source

You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.

by @TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5334

🌬️ 🍽️ Airtable Integration

Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.

by @TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5404

✂️ Add trim reformat action to transformer block

A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.

by @cristopheridlc in https://github.com/mage-ai/mage-ai/pull/5321

🐻‍❄️ Enable polars dataframe in GCS data exporter

This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.

by @TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5348

Add support to scheduler name on k8s executor

This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.

by @messerzen in https://github.com/mage-ai/mage-ai/pull/5412

🐛 Bug Fixes

• [jk] Update condition for cookie secure property by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/5327
• [xy] Fix NoneType in trigger settings. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5329
• [xy] Fix dbt block delete request with nginx. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5332
• [xy] Fix API for file versions. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5338
• [xy] Fix schema creation when schema name is wrapped with double quotes. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5366
• [jk] Update test ENV value by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/5372
• [xh] Add retry logic and throw if open AI API returns error by @matrixstone in https://github.com/mage-ai/mage-ai/pull/5375
• [jk] Fix typo for custom frequency in trigger review by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/5392
• [td] Fix Git syncs API error returning sync_on_executor_start by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/5388
• #5417 Fix bug: Can't use git with Bitbucket by @kennynguyeenx in https://github.com/mage-ai/mage-ai/pull/5418
• Dynamic block fixes
  • [td] Fix index used in dynamic block by @tommydangerous in https://github.com/mage-ai/mage-ai/pull/5373
  • [xy] Fix the condition of checking dynamic upstreams completed https://github.com/mage-ai/mage-ai/commit/2f2eeca44d1aa4555a1165bcc3c7f1bfa414c674
  • [xy] Some updates and fixes for dynamic blocks https://github.com/mage-ai/mage-ai/commit/7eba140fba59d455da89b1eef54e723b76128693
• [hw] Fix a missing import issue with data types in a template for Data Exporters by @csharplus in https://github.com/mage-ai/mage-ai/pull/5377
• SASL_SSL kafka Host Name Validation by @sujiplr in https://github.com/mage-ai/mage-ai/pull/5380

💅 Enhancements & Polish

• [xy] Reduce CPU usage in scheduler. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5331
• [xy] Pass main pod labels to workspaces. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5344
• enhancement: bump duckdb to 1.0.0 by @matsonj in https://github.com/mage-ai/mage-ai/pull/5358
• Pass all SSL settings through for SASL_SSL by @csharplus in https://github.com/mage-ai/mage-ai/pull/5381
• Update unit tests to use the methods compatible with Python 3.12 by @csharplus in https://github.com/mage-ai/mage-ai/pull/5398
• Introduce StrEnum and IntEnum classes to resolve unit test failures with Python 3.11+ by @csharplus in https://github.com/mage-ai/mage-ai/pull/5397
• Add NATS credentials file authentication by @ddecaro11 in https://github.com/mage-ai/mage-ai/pull/5410
• Update a few unit tests that failed on Python 3.12 by @csharplus in https://github.com/mage-ai/mage-ai/pull/5420
• Fix kafka module related unit test failure issues in Python 3.12 by @csharplus in https://github.com/mage-ai/mage-ai/pull/5424
• Add trigger name in kwargs https://github.com/mage-ai/mage-ai/commit/81e4f91a238d17ba152fe38246fb45b756c74882
• Expose column name related config to postgres sink config https://github.com/mage-ai/mage-ai/commit/c40af5f228491374a726b5a7b69b734b2730c81a
• [xh] Add ENV variables to turn on/off AI API by @matrixstone in https://github.com/mage-ai/mage-ai/pull/5394

Other Changes

• Update amazon-s3.mdx by @Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5337
• Fix spelling in compute-resource.mdx by @samacciu23 in https://github.com/mage-ai/mage-ai/pull/5356
• Update logger.py by @vffv2000 in https://github.com/mage-ai/mage-ai/pull/5383
• [hw] Update the user guide for Kafka streaming with latest changes based on testing by @csharplus in https://github.com/mage-ai/mage-ai/pull/5378
• Update the development document to resolve one issue encountered by @csharplus in https://github.com/mage-ai/mage-ai/pull/5396
• Fixing spelling error by @benjaminhawn in https://github.com/mage-ai/mage-ai/pull/5407

New Contributors

• @cristopheridlc made their first contribution in https://github.com/mage-ai/mage-ai/pull/5321
• @TalaatHasanin made their first contribution in https://github.com/mage-ai/mage-ai/pull/5348
• @samacciu23 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5356
• @matsonj made their first contribution in https://github.com/mage-ai/mage-ai/pull/5358
• @vffv2000 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5383
• @benjaminhawn made their first contribution in https://github.com/mage-ai/mage-ai/pull/5407
• @ddecaro11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5410
• @Jaykold made their first contribution in https://github.com/mage-ai/mage-ai/pull/5340
• @kennynguyeenx made their first contribution in https://github.com/mage-ai/mage-ai/pull/5418
• @messerzen made their first contribution in https://github.com/mage-ai/mage-ai/pull/5412

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.73...0.9.74