[Feature] Support data ingestion for range distribution table #66196
+3,046
−23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
@cursor review |
Contributor
|
@cursor review |
Contributor
🧪 CI InsightsHere's what we observed from your CI run for a007d96. 🟢 All jobs passed!But CI Insights is watching 👀 |
wyb
reviewed
Dec 5, 2025
Contributor
|
@cursor review |
a5ce2b8 to
6ebc144
Compare
Contributor
|
@cursor review |
wyb
reviewed
Dec 9, 2025
fe/fe-core/src/main/java/com/starrocks/planner/OlapTableSink.java
Outdated
Show resolved
Hide resolved
| class SlotDescriptor; | ||
|
|
||
| // RangeRouter is responsible for routing rows will ranges which represent the entire (-inf, +inf) | ||
| class RangeRouter { |
Contributor
There was a problem hiding this comment.
can we reuse the partition range code?
Contributor
Author
There was a problem hiding this comment.
The code logic is similar, but the RangeRouter is more memory-efficient. In addition, OlapTablePartitionParam contains a lot of partition-related logic, and I think decoupling it would be more appropriate for future iterations.
Contributor
|
@cursor review |
## What I'm doing: Implement end-to-end range-based routing for range distribution tables on both FE and BE sides: FE: Add TabletRange and Tuple.toThrift / Variant.toThrift to serialize tablet ranges as TTabletRange/TVariant. OlapTableSink / FrontendServiceImpl now populate range_distributed_columns and per-tablet range in TOlapTablePartitionParam and TOlapTableIndexTablets, and temporarily disable colocate MV index for RANGE distribution tables. BE: Introduce RangeRouter to route rows to tablets based on TTabletRange, supporting multi-column ranges and various open/closed/±inf intervals; Extend TabletSinkSender with RangeTabletSinkSender. Tests: Add/extend FE tests (VariantTest, TabletRangeTest) to cover numeric, string, boolean and date/datetime serialization to Thrift. Add BE tests (RangeRouterTest, TabletSinkSenderRangeTest) covering typical routing paths, boundary behavior, sparse selections and error cases. Signed-off-by: srlch <linzichao@starrocks.com>
Contributor
|
@cursor review |
🚨 Bugbot couldn't runSomething went wrong. Try again by commenting "Cursor review" or "bugbot run", or contact support (requestId: serverGenReqId_b6e0ba7e-f4b8-410c-9135-5955b40f212f). |
Contributor
|
@cursor review |
Contributor
|
@cursor review |
Contributor
|
@cursor review |
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]❌ fail : 54 / 69 (78.26%) file detail
|
[BE Incremental Coverage Report]✅ pass : 202 / 229 (88.21%) file detail
|
| // the _boundaries will be: [100, 200] | ||
| // the _lower_bound_inclusive will be: [false, true, true] | ||
| std::vector<ColumnPtr> _boundaries; | ||
| std::vector<uint8_t> _lower_bound_inclusive; |
Contributor
There was a problem hiding this comment.
The first element of _lower_bound_inclusive is always false, so it do not need to save?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I'm doing:
Implement end-to-end range-based routing for range distribution tables on both FE and BE sides:
FE:Add TabletRange and Tuple.toThrift / Variant.toThrift to serialize tablet ranges as TTabletRange/TVariant. OlapTableSink / FrontendServiceImpl now populate range_distributed_columns and per-tablet range in TOlapTablePartitionParam and TOlapTableIndexTablets, and temporarily disable colocate MV index for RANGE distribution tables.
BE:Introduce RangeRouter to route rows to tablets based on TTabletRange, supporting multi-column ranges and various open/closed/±inf intervals; Extend TabletSinkSender with RangeTabletSinkSender.
Tests:Add/extend FE tests (VariantTest, TabletRangeTest) to cover numeric, string, boolean and date/datetime serialization to Thrift. Add BE tests (RangeRouterTest, TabletSinkSenderRangeTest) covering typical routing paths, boundary behavior, sparse selections and error cases.
Fixes #64986
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
Note
Implements range-based row routing for range-distributed tables, adding FE/BE support, Thrift/Proto range serialization, and comprehensive tests.
RangeRouterandRangeTabletSinkSenderto route rows byTTabletRange; wire intoOlapTableSinkwhenis_range_distribution().OlapTablePartitionParamto carrydistribution_type; compute hashes only for HASH; randomize for RANGE/RANDOM.range_router_testandtablet_sink_sender_range_test.TabletRange,Tuple.toThrift(), andVariant.toThrift()(BOOL/INT/LARGEINT/STRING/DATE) to serialize tablet ranges.OlapTableSink/FrontendServiceImpl: populatedistribution_type, range-distributed columns, and per-tabletrangeinTOlapTablePartitionParam/TOlapTableIndexTablets; disable colocate MV for RANGE.MetaUtilsfor range distribution columns/ids;Tabletnow holdsTabletRange.VariantTest,TabletRangeTest; adjust existing tests for distribution info.TOlapTableDistributionTypeanddistribution_typetoTOlapTablePartitionParam.TOlapTableTablet.rangeandTOlapTableIndexTablets.tablets.TTabletRangenow includes bounds and inclusivity;TVariant/VariantPBuselong_valueandstring_valuefields.Written by Cursor Bugbot for commit a007d96. This will update automatically on new commits. Configure here.