apache/flink-cdc
Fork: 1941 Star: 5705 (更新于 2024-11-14 05:24:13)
license: Apache-2.0
Language: Java .
Flink CDC is a streaming data integration tool
最后发布版本: release-3.2.0 ( 2024-09-21 00:31:21)
Flink CDC is a distributed data integration tool for real time data and batch data. Flink CDC brings the simplicity and elegance of data integration via YAML to describe the data movement and transformation in a Data Pipeline.
The Flink CDC prioritizes efficient end-to-end data integration and offers enhanced functionalities such as full database synchronization, sharding table synchronization, schema evolution and data transformation.
Getting Started
- Prepare a Apache Flink cluster and set up
FLINK_HOME
environment variable. -
Download Flink CDC tar, unzip it and put jars of pipeline connector to Flink
lib
directory. - Create a YAML file to describe the data source and data sink, the following example synchronizes all tables under MySQL app_db database to Doris :
source:
type: mysql
hostname: localhost
port: 3306
username: root
password: 123456
tables: app_db.\.*
sink:
type: doris
fenodes: 127.0.0.1:8030
username: root
password: ""
transform:
- source-table: adb.web_order01
projection: \*, format('%S', product_name) as product_name
filter: addone(id) > 10 AND order_id > 100
description: project fields and filter
- source-table: adb.web_order02
projection: \*, format('%S', product_name) as product_name
filter: addone(id) > 20 AND order_id > 200
description: project fields and filter
route:
- source-table: app_db.orders
sink-table: ods_db.ods_orders
- source-table: app_db.shipments
sink-table: ods_db.ods_shipments
- source-table: app_db.products
sink-table: ods_db.ods_products
pipeline:
name: Sync MySQL Database to Doris
parallelism: 2
user-defined-function:
- name: addone
classpath: com.example.functions.AddOneFunctionClass
- name: format
classpath: com.example.functions.FormatFunctionClass
- Submit pipeline job using
flink-cdc.sh
script.
bash bin/flink-cdc.sh /path/mysql-to-doris.yaml
- View job execution status through Flink WebUI or downstream database.
Try it out yourself with our more detailed tutorial. You can also see connector overview to view a comprehensive catalog of the connectors currently provided and understand more detailed configurations.
Join the Community
There are many ways to participate in the Apache Flink CDC community. The
mailing lists are the primary place where all Flink
committers are present. For user support and questions use the user mailing list. If you've found a problem of Flink CDC,
please create a Flink jira and tag it with the Flink CDC
tag.
Bugs and feature requests can either be discussed on the dev mailing list or on Jira.
Contributing
Welcome to contribute to Flink CDC, please see our Developer Guide and APIs Guide.
License
Special Thanks
The Flink CDC community welcomes everyone who is willing to contribute, whether it's through submitting bug reports,
enhancing the documentation, or submitting code contributions for bug fixes, test additions, or new feature development.
Thanks to all contributors for their enthusiastic contributions.
最近版本更新:(数据更新于 2024-10-23 09:55:26)
2024-09-21 00:31:21 release-3.2.0
2024-06-20 14:18:38 release-3.1.1
2024-05-24 19:05:11 release-3.1.0
2024-01-22 10:16:05 release-3.0.1
2023-12-07 11:02:37 release-3.0.0
2023-10-31 23:36:34 release-2.4.2
2023-07-21 17:34:10 release-2.4.1
2023-06-25 16:02:54 release-2.4.0
2022-11-10 09:56:44 release-2.3.0
2022-04-26 20:42:52 release-2.2.1
主题(topics):
batch, cdc, change-data-capture, data-integration, data-pipeline, distributed, elt, etl, flink, kafka, mysql, paimon, postgresql, real-time, schema-evolution
apache/flink-cdc同语言 Java最近更新仓库
2024-11-21 20:32:01 spring-projects/spring-boot
2024-11-21 09:44:43 Card-Forge/forge
2024-11-19 23:42:52 spring-projects/spring-authorization-server
2024-11-18 21:40:12 4379711/lol-helper
2024-11-16 22:50:41 xuxueli/xxl-job
2024-11-16 19:06:35 Stirling-Tools/Stirling-PDF