Skip to content
This repository was archived by the owner on Oct 10, 2025. It is now read-only.

Avoid reserializing catalog/storage metadata if there are no changes since last checkpoint - #5462

Merged
royi-luo merged 7 commits into
masterfrom
royi/serialize-optimization
May 29, 2025
Merged

Avoid reserializing catalog/storage metadata if there are no changes since last checkpoint#5462
royi-luo merged 7 commits into
masterfrom
royi/serialize-optimization

Conversation

@royi-luo

@royi-luo royi-luo commented May 27, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Detects if any changes were made to the catalog/storage metadata since the last checkpoint. If not, skips serializing and reuses the same pages + serialized data from the last checkpoint (as stored in the DB header).

  • Table::checkpoint now returns a boolean that says whether there are any checkpointed changes for the current table
  • Added a version for PageManager that increments on each FSM modification. This together with the catalog version helps determine if there are any changes in the current checkpoint
  • When writing a checkpoint now reads the database header from the previous checkpoint. This data is used in case we skip serializing in the current checkpoint.

Contributor agreement

@royi-luo royi-luo self-assigned this May 27, 2025
@royi-luo
royi-luo requested a review from benjaminwinger as a code owner May 27, 2025 20:13
@royi-luo
royi-luo requested review from ray6080 and removed request for benjaminwinger May 27, 2025 20:42
@github-actions

github-actions Bot commented May 27, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmark Result

Master commit hash: f9e6c561a20f27f141a517e9d5475b3627c070a7
Branch commit hash: fe610fd52216b73d7dfbb63de1def3f45abac186

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 671.57 673.57 -1.99 (-0.30%)
aggregation q28 6507.78 6547.78 -40.00 (-0.61%)
filter q14 62.06 59.56 2.50 (4.19%)
filter q15 65.45 63.46 1.99 (3.14%)
filter q16 277.07 277.24 -0.17 (-0.06%)
filter q17 379.48 379.08 0.39 (0.10%)
filter q18 1859.17 1884.44 -25.27 (-1.34%)
filter zonemap-node 22.41 23.50 -1.09 (-4.64%)
filter zonemap-node-lhs-cast 22.54 23.72 -1.17 (-4.95%)
filter zonemap-node-null 22.43 23.37 -0.94 (-4.03%)
filter zonemap-rel 5614.12 5649.96 -35.83 (-0.63%)
fixed_size_expr_evaluator q07 627.30 617.27 10.03 (1.62%)
fixed_size_expr_evaluator q08 912.91 902.74 10.17 (1.13%)
fixed_size_expr_evaluator q09 910.10 904.87 5.23 (0.58%)
fixed_size_expr_evaluator q10 194.44 191.84 2.60 (1.36%)
fixed_size_expr_evaluator q11 194.87 191.62 3.25 (1.69%)
fixed_size_expr_evaluator q12 171.81 168.19 3.62 (2.15%)
fixed_size_expr_evaluator q13 1505.71 1497.46 8.25 (0.55%)
fixed_size_seq_scan q23 46.24 47.90 -1.66 (-3.46%)
join q29 664.72 752.67 -87.95 (-11.68%)
join q30 1512.95 1638.00 -125.05 (-7.63%)
join q31 6.98 6.45 0.53 (8.26%)
join SelectiveTwoHopJoin 45.96 44.70 1.27 (2.83%)
ldbc_snb_ic q35 10.27 10.09 0.18 (1.81%)
ldbc_snb_ic q36 87.01 96.56 -9.55 (-9.89%)
ldbc_snb_is q32 2.46 3.54 -1.08 (-30.43%)
ldbc_snb_is q33 14.10 13.60 0.50 (3.68%)
ldbc_snb_is q34 1.22 1.21 0.01 (0.63%)
limit push-down-limit-into-distinct 1926.27 2053.52 -127.25 (-6.20%)
multi-rel multi-rel-large-scan 1693.55 1662.13 31.41 (1.89%)
multi-rel multi-rel-lookup 4.41 8.62 -4.20 (-48.76%)
multi-rel multi-rel-small-scan 196.95 194.23 2.72 (1.40%)
order_by q25 81.35 65.34 16.01 (24.51%)
order_by q26 403.06 374.52 28.55 (7.62%)
order_by q27 1324.38 1348.19 -23.80 (-1.77%)
recursive_join recursive-join-bidirection 371.65 350.82 20.84 (5.94%)
recursive_join recursive-join-dense 7055.32 6924.76 130.56 (1.89%)
recursive_join recursive-join-path 23369.28 23376.84 -7.55 (-0.03%)
recursive_join recursive-join-sparse 10.42 9.48 0.95 (9.99%)
recursive_join recursive-join-trail 6955.05 6891.92 63.13 (0.92%)
scan_after_filter q01 101.95 103.56 -1.61 (-1.56%)
scan_after_filter q02 93.17 94.46 -1.29 (-1.37%)
shortest_path_ldbc100 q37 77.43 72.62 4.82 (6.63%)
shortest_path_ldbc100 q38 325.30 358.87 -33.57 (-9.35%)
shortest_path_ldbc100 q39 86.77 85.23 1.54 (1.80%)
shortest_path_ldbc100 q40 508.19 498.28 9.92 (1.99%)
var_size_expr_evaluator q03 2065.55 2035.83 29.72 (1.46%)
var_size_expr_evaluator q04 2175.44 2185.87 -10.43 (-0.48%)
var_size_expr_evaluator q05 2604.69 2525.79 78.90 (3.12%)
var_size_expr_evaluator q06 1287.58 1269.84 17.74 (1.40%)
var_size_seq_scan q19 1365.34 1345.09 20.26 (1.51%)
var_size_seq_scan q20 2569.28 2666.03 -96.75 (-3.63%)
var_size_seq_scan q21 2212.26 2165.86 46.40 (2.14%)
var_size_seq_scan q22 109.95 110.99 -1.04 (-0.94%)

@codecov

codecov Bot commented May 27, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.67%. Comparing base (f9e6c56) to head (487d4c4).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #5462   +/-   ##
=======================================
  Coverage   86.66%   86.67%           
=======================================
  Files        1418     1418           
  Lines       62184    62213   +29     
  Branches     7632     7630    -2     
=======================================
+ Hits        53890    53921   +31     
+ Misses       8115     8113    -2     
  Partials      179      179

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@royi-luo
royi-luo force-pushed the royi/serialize-optimization branch from f777d24 to 487d4c4 Compare May 29, 2025 12:43
@royi-luo
royi-luo merged commit 412d632 into master May 29, 2025
@royi-luo
royi-luo deleted the royi/serialize-optimization branch May 29, 2025 13:55
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer
@ray6080 ray6080 ray6080 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@royi-luo royi-luo

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@royi-luo @ray6080