refactor(records): drop records_batch table 3/3#6126
Merged
Conversation
This was referenced May 12, 2026
68ac5c8 to
27e3642
Compare
75a644f to
bcdddd8
Compare
27e3642 to
30612db
Compare
bcdddd8 to
0b8b9cb
Compare
30612db to
56f18fa
Compare
kaposke
approved these changes
May 13, 2026
pfreixes
approved these changes
May 13, 2026
0b8b9cb to
570adc3
Compare
pull Bot
pushed a commit
to admariner/Pizzly
that referenced
this pull request
May 13, 2026
Introduce records_seen table (daily partitions) tracked alongside records_batch so removing expired entries doesn't require deleting entries which is an expensive process that creates lots of dead tuples. Instead we can drop old partitions which is a quick operation that doesn't mutate any rows/indexes Extend batchCleanup daemon to pre-create tomorrow's partition and drop old ones on each tick. Partition table isn't read yet. part 2: NangoHQ#6125 part 3: NangoHQ#6126 <!-- Describe the problem and your solution --> <!-- Issue ticket number and link (if applicable) --> <!-- Testing instructions (skip if just adding/editing providers) -->
570adc3 to
a40cf11
Compare
TBonnin
added a commit
to calebcgates/25-pr-nango
that referenced
this pull request
May 14, 2026
…goHQ#6125) part 1: NangoHQ#6124 now using the records_seen partitioned table in deleteOutdatedRecords and remove usage of the deprecated records_batch table next PR will remove the table entirely part 3: NangoHQ#6126
…rrent partition management Introduce records_seen table (daily partitions) tracked alongside records_batch so removing expired entries doesn't require deleting entries which is an expensive process that creates lots of dead tuples. Instead we can drop old partitions. Extend batchCleanup daemon to pre-create tomorrow's partition and drop old ones on each tick. Partition table isn't read yet. Following PRs will modify the deleteOutdatedRecords to use the new partitioned table and will remove the records_batch one
now using the records_seen partitioned table in deleteOutdatedRecords and remove usage of the deprecated records_batch table next PR will remove the table entirely
…rrent partition management Introduce records_seen table (daily partitions) tracked alongside records_batch so removing expired entries doesn't require deleting entries which is an expensive process that creates lots of dead tuples. Instead we can drop old partitions. Extend batchCleanup daemon to pre-create tomorrow's partition and drop old ones on each tick. Partition table isn't read yet. Following PRs will modify the deleteOutdatedRecords to use the new partitioned table and will remove the records_batch one
now using the records_seen partitioned table in deleteOutdatedRecords and remove usage of the deprecated records_batch table next PR will remove the table entirely
records_seen fully replaces records_batch for tracking seen record ids.
56f18fa to
f2487c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part 1: #6124
part 2: #6125
records_seenfully replacesrecords_batchfor tracking seen record ids.