What's Changed

🎉 Exciting New Features

🐍 Custom python source in Streaming pipeline

This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:

from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable

if 'streaming_source' not in globals():
    from mage_ai.data_preparation.decorators import streaming_source


@streaming_source
class CustomSource(BasePythonSource):
    def init_client(self):
        """
        Implement the logic of initializing the client.
        """

    def batch_read(self, handler: Callable):
        """
        Batch read the messages from the source and use handler to process the messages.
        """
        while True:
            records = []
            # Implement the logic of fetching the records
            if len(records) > 0:
                handler(records)

by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4763

📦 OracleDB exporter

Add OracleDB exporter template to standard batch pipeline.

by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4785

🔧 Server logging format template overriding

This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems. Here is an example setting:

export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'

by @afal007 in https://github.com/mage-ai/mage-ai/pull/4772

🔄 Update pipeline name before pipeline creation

Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.

by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4790

🐛 Bug Fixes

• [jk] Add check to avoid rendering of invalid JSX element by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4755
• [dy] Fix GitLab auth issues by @dy46 in https://github.com/mage-ai/mage-ai/pull/4760
• [jk] Increase visibility of code block context menu by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4768
• [dy] Check status response for REQUIRE_USER_AUTHENTICATION by @dy46 in https://github.com/mage-ai/mage-ai/pull/4782
• [xy] Fix exporting dataframe to bigquery. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4783
• [xy] Catch job manager init error. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4784
• [dy] Add all parent block variables to the add on block by @dy46 in https://github.com/mage-ai/mage-ai/pull/4747
• [dy] Fix clickhouse destination by @dy46 in https://github.com/mage-ai/mage-ai/pull/4788
• [jk] Bugfixes - table render / rename destination block by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4797
• [dy] Update API trigger url by @dy46 in https://github.com/mage-ai/mage-ai/pull/4800
• [dy] Load pipeline tags from config by @dy46 in https://github.com/mage-ai/mage-ai/pull/4806
• [xy] Fix insert command for trino deltalake array type. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4819
• [jk] Fix unclickable dbt sql file in dbt v1 model select by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4822
• [xy] Raise ImportError when spark is not supported. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4825
• [xy] Fix mongodb source typo. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4833
• [jk] Configure dbt v2 yaml block and other dbt code block improvements by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4834
• [xy] Fix writing polars dataframe to gcs storage. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4839
• [xy] Create snowflake temp table like original table. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4802
• [jk] Bugfix pipeline tag cache by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4841
• [xh] Oracle fix issue: incremental datetime bookmark fetch fails by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4816
• [jk] Clear block_uuid query param when selecting new block by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4844
• [dy] Add sign in check to index.tsx by @dy46 in https://github.com/mage-ai/mage-ai/pull/4781
• [dy] Fix SSO sign in for frontend build by @dy46 in https://github.com/mage-ai/mage-ai/pull/4794
• [dy] Put cache in try/except by @dy46 in https://github.com/mage-ai/mage-ai/pull/4796
• [dy] Fix slack notifications with stacktrace by @dy46 in https://github.com/mage-ai/mage-ai/pull/4826
• [dy] Update config_overwrite by @dy46 in https://github.com/mage-ai/mage-ai/pull/4811

💅 Enhancements & Polish

• [xh] OracleDB thick mode support in standard pipeline by @matrixstone in https://github.com/mage-ai/mage-ai/pull/4701
• [jk] Limit number of block output files read and included in pipeline response by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4758
• [xy] Only show sql logs in debug environment. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4789
• [xy] Interpolate global vars in k8s executor namespace by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4792
• [enhancement] mongodb stream full_document config option by @moseswynn in https://github.com/mage-ai/mage-ai/pull/4793
• [xy] Improve the performance of loading dbt blocks. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4807
• [jk] Avoid N dbt config option requests for N blocks on Pipeline Editor by @johnson-mage in https://github.com/mage-ai/mage-ai/pull/4812
• [xy] Speed up backend requests. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4815
• [xy] Test kafka connection when initializing the consumer. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4832
• [xy] Make path_to_credential optional in GCS logger. by @wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4837
• [dy] Expire certain pipeline runs by @dy46 in https://github.com/mage-ai/mage-ai/pull/4727
• [dy] Add pagination to workspace roles by @dy46 in https://github.com/mage-ai/mage-ai/pull/4799
• [dy] Improve elasticsearch speed by @dy46 in https://github.com/mage-ai/mage-ai/pull/4824

✨ New Contributors

• @afal007 made their first contribution in https://github.com/mage-ai/mage-ai/pull/4772
• @moseswynn made their first contribution in https://github.com/mage-ai/mage-ai/pull/4793

Full Changelog: https://github.com/mage-ai/mage-ai/compare/0.9.66...0.9.68