Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.3

Features and Improvements

1. [mysql] Improve error message under the case that start reading from earliest but schema change happened before (#1724)
2. [mysql] Use event filter to drop events earlier than the specified timestamp in timestamp startup mode (#1724)
3. [docs][oracle] Add new introduced config options from incremental snapshot framework (#1718)
4. [docs][oceanbase][zh] Update outdated sections and translate them to Chinese
5. [hotfix][docs] Add link for Oceanbase ZH docs
6. [docs][oceanbase] Add Chinese documents for oceanbase-cdc connector(#1611)
7. [db2] Fix stream api example in db2 doc. (#1725)
8. [oracle] Add missing supported config options for oracle table source (#1720)
9. [mysql] Fix MySqlBinlogSplitAssigner may reassign split after exception restart (#1713)
10. [docs] Repair Connector Data Type links
11. [docs] Update the connector download link in the document
12. [docs] Update the flink cdc picture with supported database vendors.
13. [tidb] Fix unstable TiDB region changed test. (#1702)
14. [docs][mongodb] Add docs for MongoDB incremental source
15. [oracle][mysql] Improve the Oracle all data types test and clean up debug logs
16. [oracle] Properly support TIMESTAMP_LTZ type for oracle cdc connector
17. [oracle]support use user given column as split column
18. [oracle][minor] Clean up useless methods and classes
19. [oracle] Parallel read supports startup of startupOptions
20. [oracle] Add full type field test
21. [oracle]OracleTableSource support splitSize, fetchSize ,connectMaxRetries, connectionPoolSize and add test
22. [hotfix][oracle][db2] Update the license header
23. [cdc-base][oralce] Remove useless method of JdbcSourceFetchTaskContext
24. [cdc-base][oracle] Improve the base methods base on Oracle specific requirements
25. [oracle] Support custom url for incremental snapshot source
26. [oracle] Introduce SCAN_INCREMENTAL_SNAPSHOT_ENABLED option for OracleTableSourceFactory
27. [minor][oracle] Improve tests and code format
28. [oracle] Ignore testCheckpointAndRestore util DBZ-5245 and DBZ-4936 fix
29. [oracle][test] Add tests for Oracle incremental snapshot source
30. [oracle] Use ROWID as split column and replace limit syntax with ROWNUM
31. [oracle] Let LogMinerStreamingChangeEventSource do not rely on oracle table LOG_MINING_FLUSH
32. [oracle] Use Incremental Snapshot Framework for Oracle CDC Connector (#1079)
33. [docs] Bump Flink version to 1.16.0
34. [common] Bump Flink version to 1.16.0
35. [docs][db2] Add db2 to README.md (#1699)
36. [tidb] Checkpoint is not updated long after a task has been running (#1686)
37. [hotfix] Add method getMaxResolvedTs back to class CDCClient. (#1695)
38. [docs] Bump connector version to flink 1.15.2 in docs (#1684)
39. [tidb] Fix data lost when region changed (#1632)
40. [hotfix][docs] Correct reference link for DB2 docs (#1683)
41. [mysql] Update docs of specifying starting offset feature of MySQL CDC source
42. [hotfix][mysql] Remove unused constructor in MySqlTableSource
43. [mysql] Support starting reading binlog from earliest offset / specific offset / timestamp (#1661)
44. [mongodb] Support incremental snapshot for MongoDB CDC Connector
45. [cdc-base] Extract a higher level IncrementalSource independent of JDBC dialects
46. [hotfix][docs] Fix db2 doc download link (#1681)
47. [docs][db2] Add Db2 CDC connector doc link
48. [docs][db2] Add Db2 cdc connector docs
49. [build][db2] Add Db2 pipeline script
50. [e2e][db2] Add Db2 e2e test
51. [db2] Add sql connector for Db2 CDC (#1626)
52. [db2] Introduce Db2 cdc connector (#450)
53. [tidb] Fix data loss in tidb-cdc when switch from snapshot read to change event read (#1207)
54. [hotfix][cdc-base] Fix OffsetFactory is not initialized when restored from checkpoint (#1657)
55. [common] Bump Flink version to 1.15.2 (#1504)
56. [cdc-base] Abstract naming for variables of incremental framework (#1646)
57. [mysql] ChunkSplitter supports to snapshot when splitting chunks for a table (#1463)
58. [cdc-base] Fix StreamSplit's endingOffset of StreamSplitAssigner (#1659)
59. [cdc-base] JdbcSourceConfig can be template type of DataSourceDialect (#1639)
60. [mysql] Use the latest received meta of binlog split when request the next meta (#1479)
61. [minor] Use Collections.singletonList instead of Arrays.asList if possible (#1569)
62. [oceanbase] Add new deserialization schema with runtime converter (#1356)
63. [mysql] Avoid duplicate split requests when add new table (#1156)
64. [postgres] Upgrade org.postgresql:postgresql from 42.2.14 to 42.2.26 for CVE-2022-31197 (#1616)
65. [test][e2e] Upgrade org.postgresql:postgresql from 42.3.1 to 42.2.26 for CVE-2022-31197(#1616)
66. [hotfix][typo] Fix typo from schemaLess to schemaless
67. [cdc-base] Reduce the memory usage of JM by sharing table schemas between splits
68. [cdc-base] Release the scan fetcher thread resource after reading finished (#1619)
69. [cdc-base] Optimize pure binlog phase check logic to improve performance (#1620)
70. [cdc-base] Let all records of one snapshot split don't cross checkpoints (#1622)
71. [postgres] Introduce changelog mode configuration with upsert/all modes
72. [common] Introduce changelog mode configuration with upsert/all modes
73. [hotfix][docs] Fix docs for scan.incremental.snapshot.chunk.size (#1617)
74. [cdc-base] Adopt streaming merge during snapshotSplit scan to avoid OOM
75. [docs][oceanbase][zh] Add video tutorials that synchronizes data from OceanBase to Elasticsearch (#1604)
76. [docs][oceanbase] Add video tutorials that synchronizes data from OceanBase to Elasticsearch (#1605)
77. [mysql] Fix parse error when ALTER SQL statement contains default string value and collate. (#1544)
78. [docs][mysql] Add regex examples for multiple tables (#1564)
79. [oceanbase] Shade apache commons dependency to avoid class conflict (#1532)
80. [postgres] add geometry type support. (#1554)
81. [build] Polish CI scripts and remove useless code (#1540)
82. [mysql] fix parsing ddl error when use string default values for numeric types (#1522)
83. [mysql] Enumerate captured tables lazily (#1507)
84. [mysql] Reduce the memory usage of JM by sharing table schemas between splits (#1523)
85. [hotfix][docs] Fix comment typo in MysqlSnapshotSplit (#1513)
86. [mysql] Let all records of one snapshot split don't cross checkpoints (#1505)
87. [tests] Remove usage of deprecated EnvironmentSettings.Builder#useBlinkPlanner (#1493)
88. [cdc-base][mysql] Fix split key comparison when may contain different types (#1499)
89. [docs][mysql] Update the description of mysql server-time-zone option (#1497)
90. [mysql] Use local timezone as the default value of 'server-time-zone' option (#1407)
91. [docs][postgres] Add two frequently debezium options in Postgres connector document (#1142)
92. [mongodb] Allow mongo ARRAY to be converted to string type in Flink (#1475)
93. [hotfix][docs] Fix the page links in MySQL Chinese document (#1466)
94. [oceanbase][test] Use memory mode of obcdc and double startup timeout duration (#1474)
95. [mysql][polardbx] Stop container timely in tests (#1468)
96. [mysql][ploardbx] Support all charsets in column level (#1166)
97. [mongodb] Expose updateLookup option for stream source. (#1461)
98. [oceanbase] Use utc offset as 'server-time-zone' option's default value (#1443)
99. [mysql][minor] Skip capture newly added tables according snapshotassigner's status (#1449)
100. [polardbx][test] Bump the polardb-x image version from 2.0.1 to 2.10 (#1423)
101. [build][test] Specify surefire provider to surefire-junit47 (#1423)
102. [hotfix][docs] Fix typo in bug report doc (#1460)
103. [minor][oracle] Rename JdbcConfigurationUtil to OracleJdbcUrlUtils
104. [oracle] Support custom JDBC URL for Oracle (#1258)
105. [docs] Correct the data type of 'server-id' option to String (#1444)
106. [hotfix][test] Fix MySqlSourceITCase failure due to StreamExecutionEnvironment#getConfiguration returned unmodifiable object (#1446)
107. [test] Fix tests missed 'update_before' records if the primary keys of source and sink are different (#1451)
108. [hotfix][oracle] Fix the shell script interpreter assignment should be at first line (#1452)
109. [hotfix][tidb-cdc] TiDB CDC dealUnsignedColumnValue throw NullPointerException. (#1448)
110. [polardbx][test] Rename PolardbxSourceTCase to PolardbxSourceITCase (#1447)
111. [mysql] Enable pass custom jdbc properties for debezium mysql connection (#1217)
112. [mysql] Release the debezium reader thread resources after reading finished (#1358)
113. [oceanbase] Introduce 'table-list' option to support capture list of tables (#1369)
114. [cdc-base] Flink CDC base registers the identical history engine on multiple tasks (#1340)
115. [hotfix][mysql] Fix compile error due to merge conflict
116. [mysql] Generates multiple chunks when approximate row count is bigger than chunk size (#1193)
117. [cdc-base] Fix NPE during snpashot scan phase (#1339)
118. [docs] Improve MySQL ZH document and update the outdated links (#1435)
119. [docs] Translate mysql cdc doc to Chinese (#1391)
120. [mysql] Introduce 'scan.incremental.snapshot.chunk.key-column' option to support splitting table by custom column (#1415)
121. [mysql] Optimize pure binlog phase check logic to improve performance (#1392)
122. [oracle] Fix mining session stopped due to 'No more data to read from socket'
123. [common] Ignore retriableException to make the debezium engine retry works
124. [test][oracle] Fix unstable tests OracleE2eITCase due to reuse container
125. [tidb] Fix NPE when the specified table does not exist (#1411)
126. [docs] Add step up SQLServer Database cdc document in sqlserver-cdc.md.
127. [mongodb] Expose batch.size config option for table source. (#1387)
128. [mongodb] Example tests for MongoDBSource (#1388)
129. [build] Use correct year and copyright owner in license files (#1390)
130. [mongodb] Allow document field defined as string type in flink sql (#1385)
131. [mysql] Fix dead loop when splitting chunks. (#1107)
132. [docs] Fix a typo in sqlserver-cdc.md (#1382)
133. [docs] Fix typo in mysql-cdc.md (#1364)
134. [docs] Fix a typo in tidb-cdc.md (#1215)
135. [docs] Add all supported type mapping for mysql cdc connector (#1373)
136. [build] Build document site only if current PR modified files under docs (#1230)
137. [oracle] Support XMLType for Oracle CDC Connector (#1342)
138. [build] Bump Flink version to 1.14.4 (#1284)
139. [mysql] Adopt streaming merge during snapshotSplit scan to avoid OOM (#1219)
140. [oceanbase] Introduce more config options of log proxy and improve the oceanbase connector(#1234)
141. [docs][cdc-base] Optimize annotations (#1320)
142. [mysql] Connection pool id should contains username (#1328)
143. [tidb-cdc] Monitor tidb table lose data, which table has one more regions. (#1196)
144. [dbz] Patch for DBZ-5126 to fix memory leak problem. (#939)
145. [hotfix][docs] Fix typo in java docs. (#1305)
146. [mongodb] Permission optimization of single database is specified (#1240)
147. [hotfix][cdc-base] Fix typo 'currenFetcher' -> 'currentFetcher' (#1299)
148. [hotfix][oceanbase] Fix the default string encoding(#1294)
149. [mongodb][doc] Fix meta field collection_name of metadata sql example (#1290)
150. [mongodb] Fix UnsupportedTemporalTypeException when define Timestamp as String (#1268)
151. [mysql] Support optional numeric unsigned zerofill types (#1197)
152. [mysql] Add heartbeatInterval field to equals() and hashCode() (#1116)
153. [mysql] Catch and hold exception in chunk splitter thread and rethrow in getNext() (#1109)
154. [mysql] Use the gtid set firstly when compare the binlog offset (#761)
155. [hotfix][tidb] Fix RowKind.INSERT judgement condition (#1073)
156. [oracle] Fix duplicated records when restoration (#1100)
157. [mysql] Throw RuntimeException timely in snapshot scan phase (#1098)
158. [docs][mongodb] Update document about MongoDB database and collection regex filter (#1089)
159. [common] Apply filter on ResolvedSchema to keep physical columns only (#1092)
160. [oracle][test] Use prebuilt docker image in tests
161. [common] Bump debezium version to 1.6.4.Final
162. [cdc-base] Improve Incremental snapshot framework code style. (#1054)
163. [tidb] Test TiDB 6.0.0 and add it to supported list (#1057)
164. [mysql] Serialize isSuspended flag for MySqlBinlogSplit (#996)
165. [docs][mysql] Fix connector document link error (#1043)
166. [sqlserver] Add more supported(tested) SqlServer versions (#1050)
167. [docs] Update the supported sink vendors
168. [oceanbase] Remove useless dependency and code of test. (#1018)
169. [docs][mysql] Add note to make the tableList usage and dependencies clearer (#1019)
170. [hotfix][docs] Fix misspell of docs title
171. [docs] Add example for Scan Newly Added tables feature (#1016)
172. [hotfix][docs] Fix jar link for PolarDB-X doc. (#1012)

Download

• flink-sql-connector-db2-cdc-2.3.0.jar
• flink-sql-connector-mongodb-cdc-2.3.0.jar
• flink-sql-connector-mysql-cdc-2.3.0.jar
• flink-sql-connector-oceanbase-cdc-2.3.0.jar
• flink-sql-connector-oracle-cdc-2.3.0.jar
• flink-sql-connector-postgres-cdc-2.3.0.jar
• flink-sql-connector-sqlserver-cdc-2.3.0.jar
• flink-sql-connector-tidb-cdc-2.3.0.jar

Contributor List (sorted alphabetically)

1. 01410172
2. Amber Moe
3. Dezhi Cai
4. Enoch
5. Hang Ruan
6. He Wang
7. JiaJia
8. Jiabao Sun
9. Junwang Zhao
10. Kyle Dong
11. Leonard Xu
12. Matrix42
13. Paul Lin
14. Qingsheng Ren
15. Qishang Zhong
16. Rinka
17. Sergey Nuyanzin
18. Tigran Manasyan
19. camelus
20. dujie
21. ehui
22. empcl
23. fbad
24. gongzhongqiang
25. hehuiyuan
26. hele.kc
27. hsldymq
28. jiabao.sun
29. legendtkl
30. leixin
31. leozlliang
32. lidoudou1993
33. lincoln lee
34. lxxawfl
35. lzshlzsh
36. molsion
37. molsionmo
38. pacino
39. rookiegao
40. skylines
41. sunny
42. vanliu
43. wangminchao
44. wangxiaojing
45. xieyi888
46. yurunchuan
47. zhmin
48. 阿洋
49. 莫贤彬