[FLINK-35464] Fixes operator state backwards compatibility from CDC 3.0.x #3369
Conversation
|
@leonardBang @PatrickRen PTAL |
There was a problem hiding this comment.
@yuxiqian Thanks for the PR! The patch and test themselves looks good to me, but I'm a bit concerning about introducing new modules for each version, even patch versions (like 3.0.1). As the project evolves there must be a lot of versions in the future, so number of modules could grow very fast. Not sure if this might burden the Maven build process.
|
Thanks for @PatrickRen's comments! I too agree that it's not very elegant to package all history versions just for compatibility test, but seems to be inevitable since there's no such thing like C's macro or conditional compilation in Java. I've moved |
|
Thanks for the tips! Addressed your comments to simplify pom files. |
…CDC 3.0.x and add migration tests (apache#3369)
This closes FLINK-35441 and FLINK-35464.
Flink CDC 3.1 changes how SchemaRegistry [de]serializes state data, which causes any checkpoint states saved with earlier version could not be restored in version 3.1.0. This PR adds serialization versioning for state payloads and ensures 3.0.x state could be successfully restored.
Unfortunately 3.1.0 introduces breaking changes without bumping serialization version, so this release will be excluded from state compatibility guarantee.
Running some real savepoint restore tests with https://github.com/yuxiqian/migration-test reveals that:
Before this change: (Snapshots compiled from
release-3.1andmaster)After this change: (Snapshots compiled from
FLINK-35464-BP-3.1andFLINK-35464)