Fiml is a feature engineering library for trading and machine learning. Build features from historical market data in Python, then use the same calculations on live events in Rust. Its feature vectors can serve as inputs to models built with LightGBM, XGBoost, CatBoost, scikit-learn, and other ML libraries.
- Streaming computation - Update indicators and feature vectors as price, volume, trade, and order-book events arrive, without recalculating the entire history.
- One calculation engine - Python bindings run the Rust engine, keeping historical feature generation and live computation consistent for the same configuration and event sequence.
- Minimal allocation - Reuse output storage and borrow feature values directly in Rust without copying a vector. Construction allocates internal state; allocation costs depend on the operation and API used.
- Portable pipelines - Export feature definitions and fitted transformation parameters from Python as JSON, then rebuild the feature pipeline in Rust.
- Model preprocessing - Combine indicators with transformations such as scaling, lagged values, missing-value imputation, and PCA to produce the model's input vector.
Fiml is under active development. Before 1.0, minor releases may contain breaking changes; see the changelog for migration guidance.
Fiml turns a stream of market events into an ordered vector of model inputs:
Events → Feature extractor → Raw features → Transformations → Model inputs
- Events carry timestamped price, volume, trade, or order-book updates. Time events support global calendar features. The same event stream can come from historical data or a live feed.
- Feature definitions describe each output: the calculation, its symbol
and source, parameters such as window size, and an output ID.
FeatureExtractorSpeccollects these definitions into a reusable configuration. - Feature extractor maintains indicator history and updates raw feature
values as relevant events arrive.
FeatureExtractorexposes these values through a feature vector, with IDs identifying the outputs. - Transformations prepare raw features for a model. They can select or scale values, add lags, or apply supported preprocessing parameters fitted in Python, such as imputation values and PCA components.
- Pipeline combines the extractor and transformations.
PipelineSpecdescribes the configuration; Rust'sPipelineand Python'sModelInputPipelineexecute its shared graph and expose named output vectors. Your model integrations consume those outputs to make predictions.
Feature-extractor and pipeline specifications can be serialized as JSON and loaded in either language. A typical workflow is to define features and fit supported transformations on historical data in Python, export the pipeline, and rebuild it in Rust for live use. The JSON stores configuration and fitted transformation parameters; indicator history and live order-book state must be rebuilt by feeding events to the new runtime.
The following features are available through both the Rust and Python extractors. Window sizes, sample lags, symbols, and applicable input sources are configurable.
| Category | Supported features |
|---|---|
| Raw fields and context | Latest price or volume fields; externally supplied values retained until replaced or cleared. |
| Returns | Simple and log returns over sample lags. |
| Moving averages | Simple moving average (SMA) over time windows. Sample SMA and EMA are pipeline transformations. |
| Volatility | Rolling population standard deviation of simple returns over sample or time windows. |
| Trade activity | Rolling trade count, total trade volume, and volume-weighted average price (VWAP) over time windows. |
| Volume flow | Cumulative volume delta (CVD) over sample windows, on-balance volume (OBV) over time windows, and cumulative volume-price trend (VPT). |
| Calendar | UTC day of week and elapsed time since the first event of the day, with a configurable fixed UTC offset for day boundaries. |
| Order-book quotes | Best bid/ask prices and sizes, mid-price, spread, spread in basis points, weighted mid-price, and microprice. |
| Order-book depth | Size at a price, price or size at a chosen depth, cumulative depth through a price, price range and total size needed to reach a target size, volume between prices, and bid/ask imbalance. |
| Order-book flow | Bid/ask volume added and removed across all levels of each directly applied delta. Snapshots and replay do not contribute flow. |
Timed indicators use configurable aggregation buckets. CVD requires the trade's aggressor side; order-book features require a configured book fed with snapshots and deltas.
Rust flow keys are OrderBookBidAddedVolume, OrderBookBidRemovedVolume,
OrderBookAskAddedVolume, and OrderBookAskRemovedVolume, each with a symbol.
Add them through FeatureExtractor::builder(...).add_feature(...) or
FeatureExtractorSpec with a configured book. Feed their IDs into the usual
SMA/EMA scalar transformers; the Python flow example
shows the same artifact and snapshot/replay contract.
Pipelines support scalar transformations and sequences of fitted preprocessing stages. Supported Python estimators export their fitted parameters for execution in Rust.
| Operation | Supported transformations |
|---|---|
| Scalar operations | Copy, rename, and reorder features; scale with supplied mean and scale; smooth observed values with sample SMA/EMA; measure rolling population std/min/max/range over sample or time windows; lag values by accepted-event count. |
| Fitted scaling | StandardScaler, RobustScaler, MinMaxScaler, and MaxAbsScaler. |
| Distribution transforms | PowerTransformer with Box-Cox or Yeo-Johnson; QuantileTransformer with uniform or normal output. |
| Missing values | SimpleImputer for NaNs using mean, median, most-frequent, or constant replacement, with optional missing-value indicators. |
| Feature selection | Explicit column selection and fitted VarianceThreshold selection. |
| Dimensionality reduction | PCA, with optional whitening. |
Python fitting uses the optional fiml[sklearn] extra. See the
Python preprocessing documentation
for supported estimator options and restrictions.
Python also provides future_value
for elapsed-time training labels, separate from feature replay. Generate labels
within each instrument and chronological dataset split.
These examples target 0.4.0.
Requires Rust 1.89 or newer and a native linker for your platform. From your application's directory, add Fiml from crates.io:
cargo add fiml@0.4.0 --features serde
cargo add serde_json --features float_roundtripThe serde feature enables JSON serialization of feature and pipeline
specifications. serde_json reads and writes those specifications;
float_roundtrip preserves fitted floating-point parameters when loading them.
For a Rust-only application that does not exchange JSON, omit --features serde
and the serde_json dependency.
To install the development version from Git:
cargo add fiml --git https://github.com/pbezglasny/fiml --features serdeRequires CPython 3.12 or newer. Create a virtual environment:
python3 -m venv .venvActivate it with source .venv/bin/activate on Linux or macOS, or
.\.venv\Scripts\Activate.ps1 in Windows PowerShell. Install from PyPI:
python -m pip install "fiml[pandas]>=0.4.0,<0.5"Installing a compatible wheel requires no Rust toolchain. If pip builds from
source, it requires the Rust toolchain and linker described above.
The installation also installs NumPy.
The optional pandas extra enables DataFrame input and output; omit [pandas]
when using only the NumPy API.
To also fit supported scikit-learn transformations:
python -m pip install "fiml[pandas,sklearn]>=0.4.0,<0.5"The sklearn extra currently requires scikit-learn >=1.9,<1.10.
See the Python installation guide
for source installation, development setup, and troubleshooting.
Compute a three-sample moving average, fit a scaler in Python, and export the
pipeline for Rust. This example requires the sklearn extra from Installation.
Save this as prepare_pipeline.py and run python prepare_pipeline.py.
It writes pipeline.json in the current directory.
from pathlib import Path
import fiml
import numpy as np
from sklearn.preprocessing import StandardScaler
raw = fiml.FeatureExtractorSpec().field("BTCUSDT", source="price", id="price")
spec = fiml.PipelineSpec(raw)
smoothed = spec.add(spec.root(), fiml.ScalarStage().sma("price", window=3, output="sma3"))
scaled = spec.add(smoothed, StandardScaler())
spec.output("model", scaled)
pipeline = spec.build()
prices = np.array([100.0, 102.0, 104.0, 106.0, 108.0])
model_inputs = pipeline.fit_transform(
kind=np.full(len(prices), fiml.KIND_PRICE, dtype=np.uint8),
symbol=np.full(len(prices), pipeline.symbol("BTCUSDT"), dtype=np.int64),
timestamp=np.arange(len(prices), dtype=np.int64) * 1_000,
price=prices,
fit_mask=np.arange(len(prices)) >= 2, # Exclude the two warm-up rows.
)["model"]
assert np.isnan(model_inputs[:2]).all()
np.testing.assert_allclose(model_inputs[-1, 0], np.sqrt(1.5))
Path("pipeline.json").write_text(pipeline.to_json(), encoding="utf-8")
print(f"Final model input: {model_inputs[-1, 0]:.6f}")The scaler is fitted on the ready SMA values [102, 104, 106]. The returned
matrix has one row per event, including the initial NaN rows. In a real
training workflow, fit preprocessing only on training data and use the frozen
pipeline for validation and live inference.
Save this as src/main.rs in the Cargo application configured in Installation.
Place pipeline.json in that application's root and run cargo run there.
use fiml::{Event, PipelineSpec, Symbol};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let json = std::fs::read_to_string("pipeline.json")?;
let spec: PipelineSpec = serde_json::from_str(&json)?;
let mut pipeline = spec.build()?;
let output = pipeline.output("model").unwrap().id();
let btc = Symbol::new("BTCUSDT")?;
// Rebuild indicator history; fitted scaler parameters came from JSON.
for (index, price) in [100.0, 102.0, 104.0, 106.0, 108.0].into_iter().enumerate() {
pipeline.handle_event(Event::price(btc, price, index as i64 * 1_000))?;
if index < 2 {
assert!(pipeline.output_by_id(output).unwrap().values()[0].is_nan());
}
}
let model_input: &[f64] = pipeline.output_by_id(output).unwrap().values();
assert!((model_input[0] - 1.5_f64.sqrt()).abs() < 1e-12);
println!("Final model input: {:.6}", model_input[0]);
Ok(())
}Both programs print Final model input: 1.224745. Continue feeding new price
events to the Rust pipeline and pass its borrowed output slice to your model.
- Warm-up and missing values: With
WarmupPolicy::FullWindow, windowed features produceNaNuntil enough samples or event time have accumulated. This is the Python builders' default;FirstValueallows partial-window results. Returns always require their exact sample lag. Simple returns need a nonzero lagged value; log returns need positive current and lagged values. Missing book levels and empty VWAP windows can also produceNaNafter warm-up. - Event time: Timestamps are Unix epoch milliseconds and must be
nondecreasing per symbol across all event kinds. Different symbols may
interleave; pipeline lags count accepted events in arrival order. Within an
extractor, timed windows advance on events for their own symbol. Global
Timeevents do not advance other symbols' windows. Standalone timed indicators exposeobserve(timestamp)to advance time without adding data. Timed SMA, OBV, trade count, trade volume, VWAP, and volatility support the full signedi64timestamp range in standalone indicators, extractors, and pipelines. Buckets align down to multiples of aggregation, including negative times; they expire whenbucket_start + window <= timestamp, even when the bucket start or expiry lies outsidei64. Aggregation and window durations remain positive whole milliseconds bounded byi64::MAX, with windows an exact multiple of aggregation and subject to history capacity limits. - Elapsed-time transformations:
TransformerDefinition::lagged_timed,delta_timed,simple_return_timed, andlog_return_timedaccept a positive whole-millisecondDuration. For example,TransformerDefinition::delta_timed(FeatureId::new("microprice"), FeatureId::new("change_500ms"), Duration::from_millis(500))compares the current observation with the newest finite value at or before the source clock minus 500 ms. These also work after earlier scalar/fitted stages. Windows share history per input, aggregation, and stage. The default 1 ms aggregation is exact. Use the corresponding_with_aggregation(input, output, lag_window, aggregation)constructor for coarser history. For example, a 24-hour lag with 1-second aggregation reserves about 1.38 MB. Each bucket retains its last finite observation and its actual timestamp; intermediate observations are discarded, and lookups never select an observation after their cutoff. Lag must be a positive multiple of aggregation;max_lag / aggregation + 2cannot exceed 1,000,000 slots (about 16 MB per shared history). Larger configurations are rejected before allocation. There is no interpolation; outputs remain NaN until an eligible retained observation exists. - Trajectory transformations:
rolling_slope,rolling_path_length,rolling_positive_movement,rolling_negative_movement,rolling_direction_change_count, androlling_max_drawdowndescribe the recent path of any preceding-stage scalar. Windows count finite observations; compatible outputs share allocation-free, logarithmic history updates. Slope requires two observations, zero deltas do not count as reversals, and drawdown uses absolute input units. See definitions and examples. - Streak and state-age transformations:
positive_streak,negative_streak,observations_since_positive/negative/nonzero, andtime_since_positive/negative/nonzerosummarize persistence of signed inputs. Compose them after delta/return stages or use signed raw features. Epsilon defines an inclusive neutral interval; time ages are milliseconds. Updates use constant space and allocate nothing. See state semantics. - Input validation: Extractors and pipelines reject NaN and infinity in numeric event payloads and reject out-of-order timestamps. Standalone indicators rely on the caller to supply finite inputs and ordered time. Handle order-book update errors and supply a fresh snapshot when resynchronization is required.
- State: Processing a batch advances the same indicator history as
processing its events individually. Start a fresh runtime before replaying
an independent stream; Python's
ModelInputPipelinealso providesreset(). Loading JSON starts with empty event history while retaining fitted transformation parameters. - Output layout and precision: Use feature IDs to identify columns rather
than assuming builder-call order. Rust feature vectors use
f64; Python returnsfloat64by default and optionally casts outputs tofloat32. Named pipeline outputs contain active coordinates only. Extractor diagnostics may include reserved cells, which remainNaN. See the graph and output API. - Allocation: Built-in indicator and transformation updates reuse preallocated storage. Construction, JSON handling, Python output arrays and DataFrames, and order-book storage may allocate. Borrowing Rust output slices avoids copying their values.
The current configuration limits include:
| Limit | Scope |
|---|---|
| 512 symbol names | Per process, including the reserved global symbol. Names are ASCII-case-insensitive; slots are not reclaimed. |
| 16 compatible outputs | Per compiled indicator group, such as windows sharing the same indicator configuration. |
| 1–10,000 events | Each pipeline lag transformation's lookback. This limit does not apply to return-indicator sample lags. |
| Whole milliseconds | Timed aggregation intervals and windows; a window must be a positive multiple of its aggregation interval. |
The minimum versions are Rust 1.89 and CPython 3.12. Declared Python versions include 3.12, 3.13, and 3.14. The release workflow targets Python wheels for Linux x86-64/AArch64, macOS x86-64/AArch64, and Windows x86-64. Installation from source requires a Rust toolchain and native linker. Release artifact validation is tracked in the release checklist.
Repository tasks use just. Python and notebook tasks also require uv.
just --list # Show available tasks
just build # Build the Rust library and Python wheel (also the default)
just test # Run Rust tests, Python tests, and the maintained notebook
just test-rust # Run only Rust tests
just test-python # Run only Python tests
just marimo # Open the notebook editorRun the compiled event-to-vector throughput benchmark with:
cargo bench -p fiml --bench compiled_dispatch
# Run the workload assertions once without collecting timings:
cargo bench -p fiml --bench compiled_dispatch -- --testIt compares FeatureExtractor::handle_event and Pipeline::handle_event for
1, 4, and 16 subscribed symbols, with 1, 3, and 4 windows respectively (8–512
samples or time buckets). Benchmark IDs include raw/model output counts.
The deterministic stream mixes price, volume, trade, and clock events, plus
an unrelated symbol that still advances the global elapsed-time feature.
Extractors compute timed SMAs and trade counts; pipelines add identity,
standard scaling, sample SMA/EMA, and grouped event-lag transformations.
Construction, full-window warm-up, output assertions, and event batch creation
are outside measurement. Each timed iteration drains a prebuilt batch into a
persistent runtime, consumes outputs with black_box, and continues timestamps
from the previous batch. Throughput counts all dispatched events, including
unrelated symbols and clock ticks. Allocation guarantees remain covered by
cargo test -p fiml --test pipeline_allocations; timings have no CI threshold.
- Python guide — DataFrame and NumPy APIs, event replay, and fitted preprocessing.
- Rust examples and Python examples — runnable applications.
- Feature-extractor JSON schema and pipeline JSON schema — exported configuration formats.
- 0.2.0 migration guide — move sample averages into pipelines and re-export older JSON configurations.
- 0.4.0 release and migration — order-book update-ID ranges.
- 0.3.0 release and migration — named graph outputs, Python result dictionaries, and pipeline JSON 4.0.
Build the Rust API reference from the repository root:
cargo doc -p fiml --all-features --no-deps --openFiml is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.