Skip to content

REE - RLE Integration on Execution #8249

@pdet

Description

@pdet

The idea is to implement RLE Vectors in DuckDB and integrate with the REE representation of Arrow.

Battle Plan:

  • Decompress REE to Flat Vector.
    In this initial phase, we will support the consumption of REE to flat vectors.
  • Change storage of RLE from segment to vector.
    Currently, our RLE runs within segments. To better support RLE vectors, we will change our storage to run on vectors instead.
    As an example of the representation:
VECTOR_SIZE=4
FLAT_VECTOR=[0,0,0,1,1,1,2,2,2,3,3,3,4,4,4,4,5,5,5,5]

Runs    | 3 1 | 2 2 | 1 3 | 4 | 4 | 
Data     | 0 1 | 1 2 | 2 3 | 4 | 5 |  
Vector  | 0   | 2    | 4    | 5 |  6 | 
  • Implement RLE Vectors.
    At this point, we should decompress RLE vectors to flat vectors when using them.
  • Change REE Integration to convert to RLE (Round-Trip it).
  • Specialized code for RLE for operators that can benefit from them

Metadata

Metadata

Assignees

Labels

Needs DocumentationUse for issues or PRs that require changes in the documentationstale

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions