Skip to content

Sort adaptive encoding columns by size for out-of-band storage#10919

Merged
nicktobey merged 3 commits into
mainfrom
nicktobey-claude/nick
Apr 22, 2026
Merged

Sort adaptive encoding columns by size for out-of-band storage#10919
nicktobey merged 3 commits into
mainfrom
nicktobey-claude/nick

Conversation

@nicktobey
Copy link
Copy Markdown
Contributor

Note: This will currently change how we write rows with adaptive encoding columns. Rows written on older versions of Dolt can still be read, but writing them back will result in a new chunk hash. But since Dolt currently doesn't use adaptive encoding and Doltgres is in beta, that's acceptable.

Claude's Summary

  • BuildPermissive now sorts adaptive encoding columns by size savings (largest first) before deciding which to move out-of-band, instead of visiting columns in sequential order
  • This keeps smaller values inline when possible, producing shorter tuples when columns have varying sizes
  • Added two new test cases verifying the largest-first ordering with 2 and 3 columns of different sizes

Test plan

  • Existing TestTupleBuilderAdaptiveEncodings tests pass (including inline one of two columns)
  • New test largest value moved out-of-band first: two columns where column 0 is small and column 1 is large — verifies column 1 goes out-of-band while column 0 stays inline
  • New test three columns largest first ordering: three columns (medium, small, large) — verifies only the largest goes out-of-band
  • Full store/val package tests pass

BuildPermissive now moves the largest adaptive values out-of-band first
instead of visiting columns in order, keeping smaller values inline when
possible. This produces shorter tuples when columns vary in size.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Apr 21, 2026

@nicktobey DOLT

read_tests from_latency to_latency percent_change
covering_index_scan 0.55 0.55 0.0
groupby_scan 9.73 9.91 1.85
index_join 1.89 1.86 -1.59
index_join_scan 1.39 1.34 -3.6
index_scan 22.69 22.28 -1.81
oltp_point_select 0.26 0.26 0.0
oltp_read_only 5.09 5.09 0.0
select_random_points 0.51 0.51 0.0
select_random_ranges 0.56 0.56 0.0
table_scan 23.1 22.28 -3.55
types_table_scan 68.05 66.84 -1.78
write_tests from_latency to_latency percent_change
oltp_delete_insert 6.43 6.32 -1.71
oltp_insert 3.13 3.07 -1.92
oltp_read_write 11.24 11.04 -1.78
oltp_update_index 3.19 3.19 0.0
oltp_update_non_index 3.07 3.02 -1.63
oltp_write_only 5.99 5.99 0.0
types_delete_insert 6.91 6.91 0.0

@coffeegoddd
Copy link
Copy Markdown
Contributor

@nicktobey DOLT

comparing_percentages
100.000000 to 100.000000
version result total
072e1ff ok 5937471
version total_tests
072e1ff 5937471
correctness_percentage
100.0

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Apr 21, 2026

@nicktobey DOLT

test_name from_latency_p95 to_latency_p95 percent_change
tpcc-scale-factor-1 54.83 54.83 0.0
test_name from_server_name from_server_version from_tps to_server_name to_server_version to_tps percent_change
tpcc-scale-factor-1 dolt 11ae0d2 44.11 dolt 088695c 44.4 0.66

@coffeegoddd
Copy link
Copy Markdown
Contributor

@nicktobey DOLT

comparing_percentages
100.000000 to 100.000000
version result total
088695c ok 5937471
version total_tests
088695c 5937471
correctness_percentage
100.0

Copy link
Copy Markdown
Member

@zachmu zachmu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nicktobey nicktobey merged commit ae876f8 into main Apr 22, 2026
31 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants