Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-3.0

Downloads

Flink CDC

Flink CDC 3.0.0 (tar, asc, sha512)

The binary release can be verified by public key with fingerprint C460A307EAF2701FC9D774EF96719B746D3D0C8A.

Flink CDC Pipeline Connectors

All connectors are release in JAR and available in Maven central repository.

• Apache Doris pipeline connector 3.0.0 (jar, asc, sha1)
• MySQL pipeline connector 3.0.0 (jar, asc, sha1)
• Starrocks pipeline connector 3.0.0 (jar, asc, sha1)
• Values pipeline connector 3.0.0 (for testing purpose, jar, asc, sha1)

Flink CDC Source Connectors

All connectors are release in JAR and available in Maven central repository.

• Db2 source connector 3.0.0 (jar, asc, sha1)
• MongoDB source connector 3.0.0 (jar, asc, sha1)
• MySQL source connector 3.0.0 (jar, asc, sha1)
• OceanBase source connector 3.0.0 (jar, asc, sha1)
• Oracle source connector 3.0.0 (jar, asc, sha1)
• Postgres source connector 3.0.0 (jar, asc, sha1)
• Microsoft SQL Server source connector 3.0.0 (jar, asc, sha1)
• TiDB source connector 3.0.0 (jar, asc, sha1)
• Vitess source connector 3.0.0 (jar, asc, sha1)

Features and Improvements

1. [pipeline-connector][starrocks] Fix char/varchar length inconsistency between cdc and starrocks (#2830)
2. [cdc-cli][cdc-composer] Use "source-table" and "sink-table" as keywords of route (#2825)
3. [build][hotfix] Remove duplicate dependency (#2826)
4. [cdc-cli][cdc-dist] Support loading global config from FLINK_CDC_HOME (#2822)
5. [cdc-common][hotfix] Remove redundant prefix of pipeline options (#2821)
6. [cdc-source-connector][oceanbase][e2e] Add OceanBase e2e test case (#2521)
7. [build] Use flink-shaded-force-shading to force all submodules to generate dependency-reduced-pom.xml
8. [build] Use ${project.version} instead of ${revision} for dependency version
9. [build] Skip maven-deploy-plugin for test modules and flink-cdc-dist
10. [cdc-pipeline-connector][mysql] Fix precision problem of BIT type conversion (#2820)
11. [cdc-source-connector][mysql] fix deserialization issue on table metadata binlog event (#2682)
12. [pipeline-connector][mysql] Remove unnecessary serverTimeZone in DebeziumEventDeserializationSchema (#2816)
13. [build][hotfix] Remove useless suppressions. (#2817)
14. [cdc-source-connector][mysql] Fix attempted load of isSuspended in MySqlSplitSerializer::deserializeSplit when cdc version <= 2.2.0 (#2550)
15. [cdc-pipeline-connector][mysql] Ensure the inference of MEDIUMINT_UNSIGNED type matches INT type in table schema (#2811)
16. [cdc-connector][oceanbase] Add option for obcdc extra configs (#2543)
17. [pipeline-connector][mysql] Enable send schema change by default (#2815)
18. [3.0][cdc-runtime] Remove waitForFlushSuccess field in request handler (#2812)
19. [pipeline-connector][starrocks] Add starrocks pipeline connector (#2765)
20. [pipeline-connector][doris] add doris pipeline connector. (#2810)
21. [cdc-pipeline-connector][mysql] Ensure the precision inference of DECIMAL type matches table schema
22. [3.0][cdc-common & runtime] Remove useless flink imports
23. [cdc-pipeline-connector][mysql] Add tests for mysql pipeline connector
24. [cdc-pipeline-connector][mysql] Support more mysql types
25. [cdc-pipeleine-connecotr][mysql] Send CreateTableEvent in the mysql pipeline connector
26. [cdc-pipeline-connector][mysql] Parse the alter statement to generate SchemaChangeEvent
27. [cdc-pipeline-connector][mysql] Introduce mysql cdc pipeline DataSource
28. [cdc-cli][cdc-composer][cdc-dist] Support submitting job to general Flink environments (#2807)
29. [cdc-runtime][hotfix] Setup waitFlushSuccess before responding operator to flush. (#2805)
30. [cdc-common] Introduce "pipeline.local-time-zone" config option which help handle time zone well (#2797)
31. [cdc-common][hotfix] Remove useless flink imports (#2803)
32. [build] Let surefire plugin to keep the full stacktrace on test failure (#2804)
33. [pipeline-common] Use column name to check whether the schema contains specific column or not. (#2801)
34. [cdc-composer][route][sink] Add name for route and improve name for sink
35. [cdc-composer][hotfix] Always chain operator by default
36. [cdc-composer][sink] Use correct name for sink operator
37. [cdc-runtime] SchemaRegistry should complete the future after making the checkpoint (#2800)
38. [cdc-connector][sqlserver][tests] Fix UT errors by increasing the wait time after committing SQL (#2799)
39. [cdc-composer] Add common and runtime JAR when submit (#2795)
40. [cdc-connector][debezium] Supports conversion from binary bytes to base64 encoded string (#2435)
41. [cdc-connector][mysql] MysqlTableSource return metadata in a fixed order (#2578)
42. [cdc-connector][mysql] Fix snapshot fetch size conf does not take effect (#2766)
43. [cdc-connector][oracle] Expose ability to skip backfill in snapshot period for Oracle CDC (#2793)
44. [cdc-connector][oracle] Fix wrong table path during backfill phase which leads to potential data loss (#2707)
45. [cdc-composer][tests] Add integration tests for FlinkPipelineComposer (#2776)
46. [cdc-dist] Introduce flink-cdc.sh as the entrypoint of the Flink CDC (#2791)
47. [3.0] [flink-cdc-dist] Collect all distro resources into a TAR (#2636)
48. [cdc-connector][sqlserver] Expose ability to skip backfill in snapshot period for SqlServer CDC.(#2783)
49. [cdc-connector][mysql] Expose ability to skip backfill in snapshot period for MySQL CDC.
50. [cdc-connector][mongodb] Expose ability to skip backfill in snapshot period for Mongodb CDC.
51. [cdc-connector][postgres] Expose ability to skip backfill in snapshot period for Postgres CDC.
52. [cdc-connector][base] Support skip snapshot backfill for incremental snapshot connector
53. [cdc-connector][base][tests] Introduce snapshot phase hooks for better snapshot test
54. [cdc-composer] Add routing logic into the Flink composer
55. [cdc-cli][cdc-composer] Use sourceTable and sinkTable as keyword in route definition
56. [cdc-runtime] Add RouteFunction for routing change events
57. [cdc-common] Introduce AssertJ-style assertion system for Flink CDC data structures
58. [hotfix][mysql][tests] Avoid canceling a finished job in tests of SpecificStartingOffsetITCase (#2687)
59. [docs][zh][hotfix] Remove redundant sections from mysql-cdc documentation (#2777)
60. [mongodb][hotfix] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U (#2760) (#2760)
61. [cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. (#2762)
62. [cdc-composer] use DataSinkWriterOperatorFactory to replace all SinkWriterOperatorFactory. (#2773)
63. [cdc-pipeline-connector][values] Avoid using default as name of namespace and schema. (#2774)
64. [cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent (#2771)
65. [cdc-common] add PublicEvolving annotation to SchemaUtils (#2756)
66. [3.0][cdc-composer] Introduce pipeline.name config to support custom flink job name (#2768)
67. [cdc-base] Extract the common parts of each ScanFetchTask into AbstractScanFetcherTask.
68. [docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial-zh.md (#2763)
69. [cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator(#2761)
70. [cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
71. [cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration (#2767)
72. [docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial.md (#2764)
73. [oceanbase] Use global timestamp to start log client (#2565)
74. [cdc-pipeline-connector][values] Add event set MULTI_SPLITS_SINGLE_TABLE to mock source with multiple splits (#2737)
75. [hotfix][cdc-composer] Use correct factory to discovery DataSink (#2757)
76. [3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
77. [3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
78. [3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
79. [build][hotfix] Use the correct format for labeler.yml (#2758)
80. [build] Add github labeler for pull request
81. [docs] Add DataStream application package guidance (#2583)
82. [cdc-common] Introduce TableFilter to filter table objects from given TableId pattern (#2673)
83. [cdc-composer] Initialize context with existing config of SourceDef and SinkDef (#2749)
84. [hotfix][docs] Adjust the correct doc versions (#2679)
85. [docs][oracle] Fix oracle cdc quick start (#2727)
86. [cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
87. [cdc-common] MetadataApplier should extends Serializable
88. [cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
89. [cdc-composer] Implementation of Flink pipeline composer for chaining everything together
90. [cdc-composer] Introduce partitioning related runtime functions and translator
91. [cdc-composer] Introduce data sink and schema operator translators
92. [cdc-composer] Introduce DataSourceTranslator for composing source
93. [cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
94. [cdc-composer] Add FlinkEnvironmentUtils for adding JAR to StreamExecutionEnvironment
95. [cdc-common] Define pipeline options and schema change behaviors
96. [cdc-common] Support adding Column directly in schema builder
97. [hotfix][vitess] Fix invalidate package name
98. [cdc-pipeline-connector][values] Introduce values cdc pipeline connector
99. [3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
100. [3.0][cdc-common][minor] Add toString method for some classes.
101. [3.0][cdc-common] Add SchemaUtil to perform schema evolution (#2732)
102. [3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes (#2685)
103. [sqlserver] Fix timestamp type parse of sqlserver (#2708)
104. [3.0][cdc-runtime] Add TypeInformation for Event class (#2728)
105. [3.0][cdc-runtime][tests] Add tests for type serializers
106. [3.0][cdc-runtime][hotfix] Move type serializers to runtime module
107. [3.0][cdc-common] Introduce data type for record field in DataChangeEvent
108. [3.0][cdc-common][minor] Improve public methods for schema event
109. [3.0][cdc-common][minor] Move FlushEvent to common module
110. [mongodb] Incremental snapshot source supports comma-separated format parameter for databases and collections (#2724)
111. [3.0][cdc-common] Use built-in ConfigOption for cdc-common module (#2716)
112. [3.0][cdc-common] Add ConText for Source/Sink initialization. (#2672)
113. [3.0][cdc-common] Add TypeSerializer implementations (#2683)
114. [build] Add header copyright check for java files
115. [common] Introduce Configuration and related utilities (#2681)
116. [oracle] Skip unsupported nested table for legacy debezium source function
117. [oracle] Skip unsupported nested table for Incremental OracleSource
118. [3.0][common] Use built-in API annotations to make common module free with flink
119. [3.0][common] Introduce built-in API annotations for project
120. [build] Bump Flink version to 1.18.0 (#2463)
121. [build][tests] Enable CI tests for CDC 3.0 modules (#2678)
122. [3.0][cdc-common] Introduce internal data structures and generic implementations (#2688)
123. [composer] Connector factory and JAR discovery utilities (#2662)
124. [postgres-cdc] Add tests for latest-offset startup strategy (#2527)
125. [3.0][cdc-common] Change DataChangeEvent from interface to class, and add implementation for DataChangeEvent. (#2677)
126. [build][tests] Enable CI tests for CDC 3.0 modules
127. [3.0][cdc-runtime] add DataSinkWriterOperator to process Event (#2649)
128. [3.0][cdc-common] Add SchemaChangeEvent implementations (#2664)
129. [cli] Implementation of CLI frontend (#2660)
130. [docs] Remove flink 1.13 from supported flink versions
131. [3.0][cdc-common] Add GenericRecordData implementation for RecordData
132. [3.0][cdc-common] Keep columns in Schema in fixed order
133. [3.0][cdc-common] Use RecordData to represent before and after record.
134. [build] Introduce -Dfast property for skipping code style checks
135. [test] Bump maven-surefire-plugin version to 3.0.0-M5
136. [test] Introduce JUnit 5 and AssertJ dependencies
137. [flink-cdc-composer] Base interfaces and models for composer and pipeline definition (#2656)
138. [3.0][cdc-common] Introduce Connector Factory APIs (#2661)
139. [3.0][cdc-common] Add getTableSchema method for MetadataAccessor (#2650)
140. [3.0][cdc-common] Introduce Schema API for CDC 3.0
141. [3.0][cdc-common] Introduce Event DataSink APIs for Flink CDC 3.0
142. [3.0][cdc-common] Introduce DataSource related APIs for Flink CDC 3.0
143. [pom] Use ${revision} replace specific version. (#2623)
144. [3.0][cdc-runtime] introduce schema evolution framework API
145. [3.0][cdc-common] Introduce data types for cdc 3.0
146. [3.0][cdc-common] Introduce Event related APIs for Flink CDC 3.0
147. [3.0][kickoff] Introduce basic modules for Flink CDC 3.0
148. [mongodb] Use SampleBucketSplitStrategy for shard collection with hashed keys (#2595)
149. [vitess][hotfix] Use stable image tag for vitess docker image (#2594)
150. [oracle] Bump oracle docker image version to 19c
151. [oracle] Refactor oracle unit test
152. [build] Bump testcontainers version and polish dependency
153. [mysql] Use automatic resource management to replace 'try - finally' code block. (#2552)
154. [sqlserver] Fix get latest lsn (#2551)
155. [flink-connector-debezium] Support overriding some debezium properties in DebeziumSourceFunction (#2530)
156. [e2e] Remove db docker image after test cases are executed. (#2508)
157. [postgres] Reuse pool connection based on database in postgres cdc
158. [mysql] remove redundant source metrics processTime and emitDelay in mysql source
159. [cdc-base]remove redundant source metrics register(#2403)
160. [cdc-base] JdbcConnectionPools supports multiple JDBC sources
161. [tests][e2e] add log config to e2e tests (#2464)
162. [postgres] Not drop replication slot for stream split (#2436)
163. [postgres] Close jdbc connection after creating replication slot for stream split (#2425)
164. [test] Close resources after each test (#2438)
165. [hotfix][docs] Fix the abnormal display of pictures (#2513)
166. [cdc-base] Fix TM hangs caused by uncaught exception (#2511)
167. [postgres] Close idle readers when snapshot finished (#2400)
168. [oceanbase][ci] Move the OceanBase CI to free azure pipeline (#2506)
169. [mongodb][hotfix] Fix Mongo collection regex match logic (#2503)
170. [hotfix][mysql] Keep assigned splits in order to fix wrong meta group calculation
171. [mysql] Add finished unack splits to state for the MysqlSourceReader (#2399)
172. [mongodb][hotfix] Mongo CDC fails to capture collections with . in names (#2488)
173. [e2e][tidb] Split e2e tests into two groups to decrease the disk usage for per azure job to avoid tidb disk full error
174. [hotfix][e2e][vitess] Fix module name typo
175. [e2e] Remove flink 1.13 tests from e2e test matrix
176. [cdc-base] Fix parsing error when serializing and deserializing the table name with dot (#2443)
177. [oracle] Correct the naming error (#2405)
178. [docs] Remove unreachable images in README (#2426)
179. [docs] Add contribution guidance section in document (#2411)
180. [sqlserver] Add support bigint type as table split column (#2416)
181. [hotfix][sqlserver] Fix backfill stream task hang (#2374)
182. [hotfix][mysql] Trim default value for BIGINT and SMALLINT types when parsing MySQL DDL (#2418)
183. [sqlserver] Add table filter to speed up SqlServerSchema read (#2369)
184. [Doc] Use redo log instead of binlog in Oracle document (#2408)
185. [mysql] Filter databases that do not need to be read when discovering tables (#2160)
186. [test] Fix TiDBE2eITCase by updating the container version to avoid tikv disk full failure (#2402)
187. [test][postgres] Close postgres containers after tests and fix container reuse bug (#2394)
188. [mysql] quota the field names when executing mysql query (#2381) (#2388)
189. [test][mysql] Provide the server-time-zone setting in MySqlSourceExampleTest#testConsumingAllEvents (#2364)
190. [hotfix][mysql] Catch underlying throwable in snapshot reading.
191. [hotfix][mysql] Fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (#2384)
192. [db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure
193. [vitess] Improve build changelog mode (#2355)
194. [mongodb][tests] Improve test coverage & add class visibility annotations
195. [mongodb] Add pendingRecords and numRecordsInErrors metrics for source
196. [mongodb][hotfix] Fix SourceRecord range check (#2343)
197. [polish] Add type parameter for MySqlRecords(#2300)
198. [doc] fix errors in PostgresParallelSourceExample (#2324)
199. [docs][oracle] Add an example of Incremental Snapshot based DataStream for Oracle CDC Connector
200. [mongodb] Add disableCursorTimeout option (#2332)
201. [mongodb] [tests] Loosen E2E timeout limitations
202. [vitess] Migrate connector properties to debezium 1.9.7 (#2260)
203. [hotfix][sqlserver] Fix sqlserver monitor same table in other database (#2336)
204. [debezium] Fix DebeizumSourceFunction can not do savepoint after close (#2259)
205. [Mysql] Fix assigning duplicate snapshot splits when enable scan newly added tables (#2096)
206. [mysql] skip closing reader when the reader received the binlog split (#2261)
207. [MySQL] Optimize how to construct config table.include.list (#2274)
208. [oracle] Fix class conflict when using multi cdc sql connector
209. [hotfix][doc] Fix doc site with 404 (#2322)
210. [docs] Add connector features table to help users quickly understand the supported features of each connector
211. [mysql] Remove the finished snapshot splits for the binlog split when restoring from the checkpoint (#2292)
212. [hotfix][sqlserver] Fix sqlserver close idle and chunk key column missing (#2305)
213. [mongodb] [docs] Adds scanFullChangelog docs (#2294)
214. [mongodb] Supports scanFullChangelog feature & deprecate copyExisting option
215. [mongodb] Support MongoDB 6.0+ & update testcases
216. [postgres] Fix the slot name conflict bug (#2251) (#2281)
217. [docs] Update Postgres CDC documents for required slot.name option (#2270)
218. [build] Add vcs.xml for IDEA (#2064)
219. [docs] add docs for the mysql tables without primary keys (#2227)
220. [MySQL] Fix the startup mode exception message (#2264)
221. [oracle] Use oracle connection in context for each reader subtask (#2254) (#2258)
222. [hotfix] Update support version (#2255)
223. [hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(xx = database, schema or table) (#2212)

Contributor List (sorted alphabetically)

1. BIN
2. Dian Qi
3. EchoLee5
4. FlechazoW
5. FocusComputing
6. Hang Ruan
7. He Wang
8. Hongshun Wang
9. Jiabao Sun
10. Josh Mahonin
11. Kunni
12. Leonard Xu
13. Maciej Bryński
14. Malcolmjian
15. North.Lin
16. Paddy Gu
17. PengFei Li
18. Qingsheng Ren
19. Shawn Huang
20. Simonas Gelazevicius
21. Sting
22. Tyrantlucifer
23. TJX2014
24. Xin Gong
25. baxinyu
26. chenlei677
27. e-mhui
28. empcl
29. gongzhongqiang
30. gaotingkai
31. ice
32. joyCurry30
33. l568288g
34. lvyanquan
35. pgm-rookie
36. rookiegao
37. skylines
38. syyfffy
39. van
40. wudi
41. wuzhenhua
42. yunqingmo
43. yuxiqian
44. zhaomin
45. zyi728
46. zhangjian