To use SlateDB as a fully-featured state backend for a stream processor, we need to support three main types of state:
- Value (Latest Value, Aggregating State)
- List / Bag / Buffering
- Map
Traditionally, state-of-the-art implementations built on top of RocksDB have leveraged the MergeOperator [1] to efficiently handle types 1 and 2.
A key example is List state, where we append elements continuously until it's time to trigger a computation—like when a window fires.
For Value state, the MergeOperator can also be useful for efficiently managing aggregating states. It allows us to defer computations until we actually need the result—such as calculating a maximum value when a large window closes. This approach can be quite effective, particularly for operations where we only need the final outcome.
While this isn't always the absolute optimal method, it's a solid foundation for future iterations and improvements.
[1] RocksDB Merge Operator
To use SlateDB as a fully-featured state backend for a stream processor, we need to support three main types of state:
Traditionally, state-of-the-art implementations built on top of RocksDB have leveraged the MergeOperator [1] to efficiently handle types 1 and 2.
A key example is List state, where we append elements continuously until it's time to trigger a computation—like when a window fires.
For Value state, the MergeOperator can also be useful for efficiently managing aggregating states. It allows us to defer computations until we actually need the result—such as calculating a maximum value when a large window closes. This approach can be quite effective, particularly for operations where we only need the final outcome.
While this isn't always the absolute optimal method, it's a solid foundation for future iterations and improvements.
[1] RocksDB Merge Operator