Wingfoil is a blazingly fast, highly scalable stream processing framework designed for latency-critical use cases such as electronic trading and real-time AI systems.
It ships with a growing library of production-ready adapters covering tick stores, message buses, market protocols, and observability backends — so you can plug graphs into real data sources and sinks with a single line.
Wingfoil simplifies receiving, processing, distributing and monitoring streaming data across your entire stack.
- Fast: Ultra low latency and high throughput with an efficient DAG based execution engine.
- Backtesting: Replay historical data to backtest and optimise strategies.
- Simple and obvious to use: Define your graph of calculations; Wingfoil manages its execution.
- Adapters: production-ready integrations for iceoryx2, KDB+, Kafka, Fluvio, FIX, ZeroMQ, etcd, Prometheus, OpenTelemetry, CSV, and more.
- Multi-language: currently available as a Rust crate, python package and a TypeScript/JavaScript client.
- Graph dynamism: rewire your graph in response to incoming data.
- Async/Tokio: seamless integration, allows you to leverage async at your graph edges.
- Multi-threading: distribute graph execution across cores.
In this example we build a simple, linear pipeline with all nodes ticking in lock-step.
use wingfoil::*;
use std::time::Duration;
fn main() {
let period = Duration::from_secs(1);
ticker(period)
.count()
.map(|i| format!("hello, world {:}", i))
.print()
.run(RunMode::RealTime, RunFor::Duration(period*3)
);
}This output is produced:
hello, world 1
hello, world 2
hello, world 3
Wingfoil lets you easily wire up complex business logic, splitting and recombining streams, and modulating the frequency of data. Adapters make it easy to plug in real data sources and sinks. In this example we load a CSV of AAPL limit orders, maintain an order book using the lobster crate, derive trades and two-way prices, and export back to CSV — all in a few lines:
let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);
let (fills, prices) = csv_read("aapl.csv", get_time, true)
.map(move |chunk| process_orders(chunk, &book))
.split();
let prices_export = prices
.filter_none()
.distinct()
.csv_write("prices.csv");
let fills_export = fills.csv_write("fills.csv");
Graph::new(vec![prices_export, fills_export], RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)
.print()
.run()
.unwrap();This output is produced:
Short code snippets for each adapter live in the examples README. The examples below are all runnable — see each one's README.md for setup and commands.
| Example | Description |
|---|---|
order_book |
Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, export to CSV. |
breadth_first |
Why wingfoil's BFS execution avoids the O(2^N) node explosion of naive depth-first DAGs. |
run_mode |
Swap RunMode::RealTime and RunMode::HistoricalFrom with the same graph wiring for backtesting. |
async |
Integrate Tokio async/await at graph edges (adapters) while keeping the core graph synchronous. |
threading |
Distribute graph execution across worker threads with producer() / mapper(). |
dynamic |
Add and remove nodes at runtime. Includes demux, dynamic-group, and dynamic-manual variants. |
feedback |
Close a loop between two nodes with a feedback channel — a proportional control loop where the plant's output feeds back into the controller's input, which a plain DAG can't express. |
tracing |
Instrumentation modes (log, tracing, instruments) for event and span handling. |
latency |
Per-hop latency stamping with Traced<T, L> and LatencyReport, transported over iceoryx2. |
| Example | Description |
|---|---|
kdb |
KDB+ integration: time-sliced reads, cached reads (LRU file cache), and round-trip write/read/validate. |
kafka |
Kafka / Redpanda adapter — subscribe, transform, publish pipeline via rdkafka. |
fluvio |
Fluvio distributed streaming — subscribe, transform, publish pipeline. |
fix |
FIX 4.4 protocol: self-contained loopback, client, echo server, and live LMAX market data over TLS. |
zmq |
ZeroMQ pub/sub with direct addressing or etcd-based service discovery. |
etcd |
etcd key-value store adapter for sub/pub with transformation. |
redis |
Redis adapter — Pub/Sub channels (subscribe, transform, republish) and persistent Streams (snapshot + tail). |
postgres |
PostgreSQL adapter — time-sliced historical reads and streaming writes, round-trip write/read/validate. |
iceoryx2 |
Zero-copy IPC over shared memory (spin, threaded, signaled polling modes). |
aeron |
Low-latency Aeron UDP/IPC transport — publish and subscribe to i64 values with spin and threaded polling modes. |
web |
WebSocket adapter streaming synthetic prices and receiving UI events. |
telemetry |
Metrics export via Prometheus scraping (pull) and OpenTelemetry OTLP (push). |
augurs |
augurs time-series toolkit — on-graph forecasting (ETS/MSTL), outlier detection (MAD/DBSCAN), changepoint, seasonality, DTW and clustering over sliding windows. |
statistics |
Streaming statistics toolkit — EWMA (per-tick and time-decayed), cumulative and rolling mean/variance/std/min/max/median, with count- and time-weighted variants over sample- and time-based windows. |
- Checkout the examples
- Download from crates.io
- Read the documentation
- Review the benchmarks
- Download the wingfoil Python module from pypi.org
- Download the
@wingfoil/clientbrowser client from npmjs.com
We want to hear from you! Especially if you:
- are interested in contributing
- know of a project that wingfoil would be well-suited for
- would like to request a feature or report a bug
- have any feedback
Please do get in touch:
- ping us on discord
- email us at hello@wingfoil.io
- submit an issue
- get involved in the discussion