Skip to content

Conversation

@tammam-g
Copy link
Contributor

@tammam-g tammam-g commented Jun 12, 2025

Fixes transactions issues by updating to the latest version and removing the extended query patching because the upstream issue was fixed (electric-sql/pglite#294)

Updating pglite to 0.3.x also meant upgrading to Postgres 17 under the hood, which makes old data directories incompatible. To address this, we've added some automatic migration code based on https://pglite.dev/docs/upgrade

Fixes #8679 and #8658

Scenarios Tested

I grabbed a old data directory, ran cat myData/PG_VERSION to confirm it was PG16, and then ran this code against it and confirmed that it migrated and started up correctly. I then reran cat myData/PG_VERSION to confirm it had been upgrade to PG17

Code Samples

image image image

@tammam-g tammam-g requested a review from joehan June 12, 2025 23:14
* Automaticly migrate pg16 to pg17

* Formatting and cleanup

---------

Co-authored-by: Joe Hanley <joehanley@google.com>
Copy link
Contributor

@joehan joehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I coauthored.

Copy link
Member

@yuchenshi yuchenshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good except one compatibility concern inline

await tempOldDb.close();
await oldDb.close();

// 4. Nuke old data directory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this step last so that if something goes wrong during the import, the developer can still go back to an older CLI version?

Even better, what if we create a subdirectory in dataDir, say pg17 and only write to that subdirectory starting from this PR? In this way, the old and new CLI versions should not mess with each other. Ever time we change the on-disk format we can move to a different subdir name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good call, implemented it:
When we migrate you, we'll copy your old dataDir over to dataDir/pg16. Then, we will use dataDir/pg17 going forward.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks great. I think we can make it even better by leaving existing data prefixless instead of pg16. In this way, downgrades are 100% safe out of the box without changing the dataDir settings.

@github-project-automation github-project-automation bot moved this from Approved [PR] to Changes Requested [PR] in [Cloud] Extensions + Functions Jun 16, 2025
@joehan joehan requested a review from yuchenshi June 16, 2025 20:55
@github-project-automation github-project-automation bot moved this from Changes Requested [PR] to Approved [PR] in [Cloud] Extensions + Functions Jun 16, 2025
@joehan joehan enabled auto-merge (squash) June 17, 2025 13:26
@joehan joehan merged commit 6dd4a51 into master Jun 17, 2025
50 checks passed
@joehan joehan deleted the tammam/fdc-emulator-transaction-fix branch June 17, 2025 13:37
@github-project-automation github-project-automation bot moved this from Approved [PR] to Done in [Cloud] Extensions + Functions Jun 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to upsert / upsertMany in DataConnect Emulator

3 participants