-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Closed
Labels
Needs DocumentationUse for issues or PRs that require changes in the documentationUse for issues or PRs that require changes in the documentationstale
Description
The idea is to implement RLE Vectors in DuckDB and integrate with the REE representation of Arrow.
Battle Plan:
- Decompress REE to Flat Vector.
In this initial phase, we will support the consumption of REE to flat vectors. - Change storage of RLE from segment to vector.
Currently, our RLE runs within segments. To better support RLE vectors, we will change our storage to run on vectors instead.
As an example of the representation:
VECTOR_SIZE=4
FLAT_VECTOR=[0,0,0,1,1,1,2,2,2,3,3,3,4,4,4,4,5,5,5,5]
Runs | 3 1 | 2 2 | 1 3 | 4 | 4 |
Data | 0 1 | 1 2 | 2 3 | 4 | 5 |
Vector | 0 | 2 | 4 | 5 | 6 |
- Implement RLE Vectors.
At this point, we should decompress RLE vectors to flat vectors when using them. - Change REE Integration to convert to RLE (Round-Trip it).
- Specialized code for RLE for operators that can benefit from them
Metadata
Metadata
Assignees
Labels
Needs DocumentationUse for issues or PRs that require changes in the documentationUse for issues or PRs that require changes in the documentationstale