Volga is a split-path dataflow engine: one SQL can run in streaming, batch (regular dataflow), or a serving-optimized request mode.

Split-path request mode: the planner cuts the dataflow graph into a write path (streaming workers maintain intermediate results in shared state) and a read path (request workers use those intermediate results to produce the final result on demand) — optimized read/write throughput and materialized state size, with a single query and no extra infrastructure.

One system (Rust, Apache Arrow, Apache DataFusion), consistent logic and execution semantics, SQL everywhere — no DSL.

Check the blog, join Slack.

Teams that run streaming pipelines almost always also need to serve the same logic (keyed lookup, often as-of a time), backfill it, and/or derive offline data over history. Today that means multiple systems: a stream processor (Flink / Spark Streaming), a serving store (Redis / KV), a batch engine, and often an orchestration layer (Chronon or custom) on top. Definitions and execution drift. Serving is a second product.

That stitch is most painful on entity-centric workloads — fraud, payments, ads, recsys, search, RAG — long per-key windows where online must match offline, streaming vs point joins, sub-optimal freshness and latency and so on. Volga is one engine for all three: stream, serve, backfill.

• One SQL, three modes. Streaming, request, and batch share operator logic. SQL compiles to a distributed dataflow graph running on master + workers, based on execution mode.

• Split-path request mode. The same query is cut into a live write path (maintain intermediate results in shared remote state) and a live read path (precompiled residual). No Redis, no second pipeline, no second SQL over a materialized view.

• Exactly-once stateful event-time processing. Event-time watermarks and windows; Chandy-Lamport barrier snapshots for recovery.

• Stateless workers. Compute–storage separation: workers do not own operator state. Operators talk to store contracts (load, atomic commit, coherent snapshot, checkpoint). Operator-specific state data structures allow for selective I/O for optimized execution; backends own partitioning, fencing, etc. Lose a worker, attach another to the same store. Checkpoints are small state versions, not a large local RocksDB dump moving over the network - fast checkpointing and failover.

• Pluggable state backends. Same contracts, different backends:

  Backend	Role
  In-memory	ships today; streaming-only
  Scylla (Cassandra-class)	latency-sensitive scenarios (streaming + request)
  SlateDB + object storage	latency-tolerant scenarios with large state (streaming + request)

• Optimized continuous sliding windows. Tiles (window-specific intermediate results) store pre-computed aggregates, allowing for long (days, months, years) read/write-time aggregates without full recompute.

• Extended native aggregates. First-class in SQL, not app UDFs: top, topn_frequency, top1_ratio, plus categorical / conditional forms (sum_cate, count_where, sum_cate_where, …).

• Kubernetes-native. kubevolga Go operator and VolgaPipeline CRD.

• Rust + Arrow + DataFusion. Columnar execution. Runs on Tokio. Actor model with kameo.

Request mode is a generic SQL write/read cut: bindings (entity key, as-of T, request row) decide where the graph splits. Streaming and batch keep the full graph.

Today the cut is a simple single window: the planner finds the top window, the window operator becomes state-only (write path), a window-request operator does the lookup (read path). Tiles are those intermediate results — an I/O and CPU optimization for long RANGE frames. Raw rows stay authoritative; tiles are not a second source of truth.

Joins are the next candidate. A general cut optimizer for an arbitrary SQL (using query shape, runtime key stats, freshness, and so on) is an area of research.

Volga is a general SQL dataflow (streaming and batch). Request is the same SQL with the plan split, so serving is not a second cluster. Compute–storage split means workers do not own the data.

Stream it continuously, or serve the same query as point lookups:

SELECT
  user_id,
  count(*) OVER w_short AS purchases_1h,
  sum(amount) OVER w_long AS spent_30d,
  top(product_id, 3) OVER w_long AS top_products_30d,
  topn_frequency(product_id, 3) OVER w_long AS top_product_freq_30d,
  sum_cate(amount, product_type) OVER w_long AS spent_per_category,
  count_cate_where(product_id, product_type, product_type = 'ON_SALE')
    OVER w_short AS on_sale_count_1h
FROM events
WINDOW
  w_short AS (
    PARTITION BY user_id
    ORDER BY event_time
    RANGE BETWEEN INTERVAL '1 hour' PRECEDING AND CURRENT ROW
  ),
  w_long AS (
    PARTITION BY user_id
    ORDER BY event_time
    RANGE BETWEEN INTERVAL '30 day' PRECEDING AND CURRENT ROW
  )

Besides standard SQL aggregates, Volga supports top / topn_frequency / top1_ratio and categorical / conditional forms (sum_cate, count_where, sum_cate_where, …).

Pipelines are JSON (or a Kubernetes VolgaPipeline CRD).

{
  "execution_mode": "Streaming",
  "parallelism": 2,
  "sources": [
    {
      "table_name": "events",
      "schema_json": {
        "fields": [
          { "name": "user_id", "type": { "name": "utf8" }, "nullable": false, "children": [] },
          { "name": "event_time", "type": { "name": "timestamp", "unit": "MILLISECOND" }, "nullable": false, "children": [] }
        ]
      },
      "source": {
        "Kafka": {
          "bootstrap_servers": "localhost:9092",
          "topic": "events",
          "offset": "Latest",
          "poll_timeout_ms": 100
        }
      }
    }
  ],
  "sql": "SELECT user_id, count(*) OVER w AS n FROM events WINDOW w AS (PARTITION BY user_id ORDER BY event_time RANGE BETWEEN INTERVAL '1 hour' PRECEDING AND CURRENT ROW)"
}

Embedded Rust uses PipelineSpec / PipelineSpecBuilder and compile_logical_graph:

use volga::api::{compile_logical_graph, PipelineSpecBuilder};

let spec = PipelineSpecBuilder::new()
    .with_parallelism(2)
    .sql("SELECT word, COUNT(*) as count FROM events GROUP BY word")
    .build();

let graph = compile_logical_graph(&spec, None);

git clone https://github.com/volga-project/volga
cd volga
cargo build

Use scripts/test for profiles, Docker/Kube setup, filters, and stress runs.

scripts/test default
scripts/test kube
scripts/test stress --env kube --all --runs-per-shard 10 --shards 2 --fresh-cluster

docker build -t volga:latest .
docker compose up

Uses docker-compose.yaml and docker/pipeline_spec.demo.json. Kubernetes: kubevolga operator + Kind + sample VolgaPipeline CR.

Volga is in active development. Streaming and request execution, RANGE windows, tiling, distributed master/worker, checkpoint failure/restore with in-memory state, event-time processing with watermarks, Kubernetes operator are real. Scylla and SlateDB backends, batch mode, backfill, regular aggregations (GROUP BY) and joins are on the roadmap.

• Scylla backend
• SlateDB + object storage backend
• Batch execution mode
• Backfills
• Regular aggregations (GROUP BY - streaming + request + split optimizer)
• Joins (streaming / temporal + request / lookup + split optimizer)
• Generic split optimizer for arbitrary SQL
• Python client
• UI
• Proper testing (correctness via sqllogictest, deterministic simulation)