Skip to content
pbezglasnyPublic

About

Streaming financial indicators and feature pipelines for machine learning, with a shared Rust engine and Python bindings.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Fiml

Русская документация

Description

Fiml is a feature engineering library for trading and machine learning. Build features from historical market data in Python, then use the same calculations on live events in Rust. Its feature vectors can serve as inputs to models built with LightGBM, XGBoost, CatBoost, scikit-learn, and other ML libraries.

  • Streaming computation - Update indicators and feature vectors as price, volume, trade, and order-book events arrive, without recalculating the entire history.
  • One calculation engine - Python bindings run the Rust engine, keeping historical feature generation and live computation consistent for the same configuration and event sequence.
  • Minimal allocation - Reuse output storage and borrow feature values directly in Rust without copying a vector. Construction allocates internal state; allocation costs depend on the operation and API used.
  • Portable pipelines - Export feature definitions and fitted transformation parameters from Python as JSON, then rebuild the feature pipeline in Rust.
  • Model preprocessing - Combine indicators with transformations such as scaling, lagged values, missing-value imputation, and PCA to produce the model's input vector.

Fiml is under active development. Before 1.0, minor releases may contain breaking changes; see the changelog for migration guidance.

Overview

Fiml turns a stream of market events into an ordered vector of model inputs:

Events → Feature extractor → Raw features → Transformations → Model inputs
  • Events carry timestamped price, volume, trade, or order-book updates. Time events support global calendar features. The same event stream can come from historical data or a live feed.
  • Feature definitions describe each output: the calculation, its symbol and source, parameters such as window size, and an output ID. FeatureExtractorSpec collects these definitions into a reusable configuration.
  • Feature extractor maintains indicator history and updates raw feature values as relevant events arrive. FeatureExtractor exposes these values through a feature vector, with IDs identifying the outputs.
  • Transformations prepare raw features for a model. They can select or scale values, add lags, or apply supported preprocessing parameters fitted in Python, such as imputation values and PCA components.
  • Pipeline combines the extractor and transformations. PipelineSpec describes the configuration; Rust's Pipeline and Python's ModelInputPipeline execute its shared graph and expose named output vectors. Your model integrations consume those outputs to make predictions.

Feature-extractor and pipeline specifications can be serialized as JSON and loaded in either language. A typical workflow is to define features and fit supported transformations on historical data in Python, export the pipeline, and rebuild it in Rust for live use. The JSON stores configuration and fitted transformation parameters; indicator history and live order-book state must be rebuilt by feeding events to the new runtime.

Supported features

Indicators and market features

The following features are available through both the Rust and Python extractors. Window sizes, sample lags, symbols, and applicable input sources are configurable.

Category Supported features
Raw fields and context Latest price or volume fields; externally supplied values retained until replaced or cleared.
Returns Simple and log returns over sample lags.
Moving averages Simple moving average (SMA) over time windows. Sample SMA and EMA are pipeline transformations.
Volatility Rolling population standard deviation of simple returns over sample or time windows.
Trade activity Rolling trade count, total trade volume, and volume-weighted average price (VWAP) over time windows.
Volume flow Cumulative volume delta (CVD) over sample windows, on-balance volume (OBV) over time windows, and cumulative volume-price trend (VPT).
Calendar UTC day of week and elapsed time since the first event of the day, with a configurable fixed UTC offset for day boundaries.
Order-book quotes Best bid/ask prices and sizes, mid-price, spread, spread in basis points, weighted mid-price, and microprice.
Order-book depth Size at a price, price or size at a chosen depth, cumulative depth through a price, price range and total size needed to reach a target size, volume between prices, and bid/ask imbalance.
Order-book flow Bid/ask volume added and removed across all levels of each directly applied delta. Snapshots and replay do not contribute flow.

Timed indicators use configurable aggregation buckets. CVD requires the trade's aggressor side; order-book features require a configured book fed with snapshots and deltas.

Rust flow keys are OrderBookBidAddedVolume, OrderBookBidRemovedVolume, OrderBookAskAddedVolume, and OrderBookAskRemovedVolume, each with a symbol. Add them through FeatureExtractor::builder(...).add_feature(...) or FeatureExtractorSpec with a configured book. Feed their IDs into the usual SMA/EMA scalar transformers; the Python flow example shows the same artifact and snapshot/replay contract.

Transformations

Pipelines support scalar transformations and sequences of fitted preprocessing stages. Supported Python estimators export their fitted parameters for execution in Rust.

Operation Supported transformations
Scalar operations Copy, rename, and reorder features; scale with supplied mean and scale; smooth observed values with sample SMA/EMA; measure rolling population std/min/max/range over sample or time windows; lag values by accepted-event count.
Fitted scaling StandardScaler, RobustScaler, MinMaxScaler, and MaxAbsScaler.
Distribution transforms PowerTransformer with Box-Cox or Yeo-Johnson; QuantileTransformer with uniform or normal output.
Missing values SimpleImputer for NaNs using mean, median, most-frequent, or constant replacement, with optional missing-value indicators.
Feature selection Explicit column selection and fitted VarianceThreshold selection.
Dimensionality reduction PCA, with optional whitening.

Python fitting uses the optional fiml[sklearn] extra. See the Python preprocessing documentation for supported estimator options and restrictions.

Python also provides future_value for elapsed-time training labels, separate from feature replay. Generate labels within each instrument and chronological dataset split.

Installation

These examples target 0.4.0.

Rust

Requires Rust 1.89 or newer and a native linker for your platform. From your application's directory, add Fiml from crates.io:

cargo add fiml@0.4.0 --features serde
cargo add serde_json --features float_roundtrip

The serde feature enables JSON serialization of feature and pipeline specifications. serde_json reads and writes those specifications; float_roundtrip preserves fitted floating-point parameters when loading them. For a Rust-only application that does not exchange JSON, omit --features serde and the serde_json dependency.

To install the development version from Git:

cargo add fiml --git https://github.com/pbezglasny/fiml --features serde

Python

Requires CPython 3.12 or newer. Create a virtual environment:

python3 -m venv .venv

Activate it with source .venv/bin/activate on Linux or macOS, or .\.venv\Scripts\Activate.ps1 in Windows PowerShell. Install from PyPI:

python -m pip install "fiml[pandas]>=0.4.0,<0.5"

Installing a compatible wheel requires no Rust toolchain. If pip builds from source, it requires the Rust toolchain and linker described above. The installation also installs NumPy. The optional pandas extra enables DataFrame input and output; omit [pandas] when using only the NumPy API.

To also fit supported scikit-learn transformations:

python -m pip install "fiml[pandas,sklearn]>=0.4.0,<0.5"

The sklearn extra currently requires scikit-learn >=1.9,<1.10. See the Python installation guide for source installation, development setup, and troubleshooting.

Simple example

Compute a three-sample moving average, fit a scaler in Python, and export the pipeline for Rust. This example requires the sklearn extra from Installation.

Python: fit and export

Save this as prepare_pipeline.py and run python prepare_pipeline.py. It writes pipeline.json in the current directory.

from pathlib import Path

import fiml
import numpy as np
from sklearn.preprocessing import StandardScaler

raw = fiml.FeatureExtractorSpec().field("BTCUSDT", source="price", id="price")
spec = fiml.PipelineSpec(raw)
smoothed = spec.add(spec.root(), fiml.ScalarStage().sma("price", window=3, output="sma3"))
scaled = spec.add(smoothed, StandardScaler())
spec.output("model", scaled)
pipeline = spec.build()

prices = np.array([100.0, 102.0, 104.0, 106.0, 108.0])
model_inputs = pipeline.fit_transform(
    kind=np.full(len(prices), fiml.KIND_PRICE, dtype=np.uint8),
    symbol=np.full(len(prices), pipeline.symbol("BTCUSDT"), dtype=np.int64),
    timestamp=np.arange(len(prices), dtype=np.int64) * 1_000,
    price=prices,
    fit_mask=np.arange(len(prices)) >= 2,  # Exclude the two warm-up rows.
)["model"]

assert np.isnan(model_inputs[:2]).all()
np.testing.assert_allclose(model_inputs[-1, 0], np.sqrt(1.5))
Path("pipeline.json").write_text(pipeline.to_json(), encoding="utf-8")
print(f"Final model input: {model_inputs[-1, 0]:.6f}")

The scaler is fitted on the ready SMA values [102, 104, 106]. The returned matrix has one row per event, including the initial NaN rows. In a real training workflow, fit preprocessing only on training data and use the frozen pipeline for validation and live inference.

Rust: load and replay

Save this as src/main.rs in the Cargo application configured in Installation. Place pipeline.json in that application's root and run cargo run there.

use fiml::{Event, PipelineSpec, Symbol};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let json = std::fs::read_to_string("pipeline.json")?;
    let spec: PipelineSpec = serde_json::from_str(&json)?;
    let mut pipeline = spec.build()?;
    let output = pipeline.output("model").unwrap().id();
    let btc = Symbol::new("BTCUSDT")?;

    // Rebuild indicator history; fitted scaler parameters came from JSON.
    for (index, price) in [100.0, 102.0, 104.0, 106.0, 108.0].into_iter().enumerate() {
        pipeline.handle_event(Event::price(btc, price, index as i64 * 1_000))?;
        if index < 2 {
            assert!(pipeline.output_by_id(output).unwrap().values()[0].is_nan());
        }
    }

    let model_input: &[f64] = pipeline.output_by_id(output).unwrap().values();
    assert!((model_input[0] - 1.5_f64.sqrt()).abs() < 1e-12);
    println!("Final model input: {:.6}", model_input[0]);
    Ok(())
}

Both programs print Final model input: 1.224745. Continue feeding new price events to the Rust pipeline and pass its borrowed output slice to your model.

Behavior and limits

  • Warm-up and missing values: With WarmupPolicy::FullWindow, windowed features produce NaN until enough samples or event time have accumulated. This is the Python builders' default; FirstValue allows partial-window results. Returns always require their exact sample lag. Simple returns need a nonzero lagged value; log returns need positive current and lagged values. Missing book levels and empty VWAP windows can also produce NaN after warm-up.
  • Event time: Timestamps are Unix epoch milliseconds and must be nondecreasing per symbol across all event kinds. Different symbols may interleave; pipeline lags count accepted events in arrival order. Within an extractor, timed windows advance on events for their own symbol. Global Time events do not advance other symbols' windows. Standalone timed indicators expose observe(timestamp) to advance time without adding data. Timed SMA, OBV, trade count, trade volume, VWAP, and volatility support the full signed i64 timestamp range in standalone indicators, extractors, and pipelines. Buckets align down to multiples of aggregation, including negative times; they expire when bucket_start + window <= timestamp, even when the bucket start or expiry lies outside i64. Aggregation and window durations remain positive whole milliseconds bounded by i64::MAX, with windows an exact multiple of aggregation and subject to history capacity limits.
  • Elapsed-time transformations: TransformerDefinition::lagged_timed, delta_timed, simple_return_timed, and log_return_timed accept a positive whole-millisecond Duration. For example, TransformerDefinition::delta_timed(FeatureId::new("microprice"), FeatureId::new("change_500ms"), Duration::from_millis(500)) compares the current observation with the newest finite value at or before the source clock minus 500 ms. These also work after earlier scalar/fitted stages. Windows share history per input, aggregation, and stage. The default 1 ms aggregation is exact. Use the corresponding _with_aggregation(input, output, lag_window, aggregation) constructor for coarser history. For example, a 24-hour lag with 1-second aggregation reserves about 1.38 MB. Each bucket retains its last finite observation and its actual timestamp; intermediate observations are discarded, and lookups never select an observation after their cutoff. Lag must be a positive multiple of aggregation; max_lag / aggregation + 2 cannot exceed 1,000,000 slots (about 16 MB per shared history). Larger configurations are rejected before allocation. There is no interpolation; outputs remain NaN until an eligible retained observation exists.
  • Trajectory transformations: rolling_slope, rolling_path_length, rolling_positive_movement, rolling_negative_movement, rolling_direction_change_count, and rolling_max_drawdown describe the recent path of any preceding-stage scalar. Windows count finite observations; compatible outputs share allocation-free, logarithmic history updates. Slope requires two observations, zero deltas do not count as reversals, and drawdown uses absolute input units. See definitions and examples.
  • Streak and state-age transformations: positive_streak, negative_streak, observations_since_positive/negative/nonzero, and time_since_positive/negative/nonzero summarize persistence of signed inputs. Compose them after delta/return stages or use signed raw features. Epsilon defines an inclusive neutral interval; time ages are milliseconds. Updates use constant space and allocate nothing. See state semantics.
  • Input validation: Extractors and pipelines reject NaN and infinity in numeric event payloads and reject out-of-order timestamps. Standalone indicators rely on the caller to supply finite inputs and ordered time. Handle order-book update errors and supply a fresh snapshot when resynchronization is required.
  • State: Processing a batch advances the same indicator history as processing its events individually. Start a fresh runtime before replaying an independent stream; Python's ModelInputPipeline also provides reset(). Loading JSON starts with empty event history while retaining fitted transformation parameters.
  • Output layout and precision: Use feature IDs to identify columns rather than assuming builder-call order. Rust feature vectors use f64; Python returns float64 by default and optionally casts outputs to float32. Named pipeline outputs contain active coordinates only. Extractor diagnostics may include reserved cells, which remain NaN. See the graph and output API.
  • Allocation: Built-in indicator and transformation updates reuse preallocated storage. Construction, JSON handling, Python output arrays and DataFrames, and order-book storage may allocate. Borrowing Rust output slices avoids copying their values.

The current configuration limits include:

Limit Scope
512 symbol names Per process, including the reserved global symbol. Names are ASCII-case-insensitive; slots are not reclaimed.
16 compatible outputs Per compiled indicator group, such as windows sharing the same indicator configuration.
1–10,000 events Each pipeline lag transformation's lookback. This limit does not apply to return-indicator sample lags.
Whole milliseconds Timed aggregation intervals and windows; a window must be a positive multiple of its aggregation interval.

Version and platform support

The minimum versions are Rust 1.89 and CPython 3.12. Declared Python versions include 3.12, 3.13, and 3.14. The release workflow targets Python wheels for Linux x86-64/AArch64, macOS x86-64/AArch64, and Windows x86-64. Installation from source requires a Rust toolchain and native linker. Release artifact validation is tracked in the release checklist.

Development

Repository tasks use just. Python and notebook tasks also require uv.

just --list       # Show available tasks
just build        # Build the Rust library and Python wheel (also the default)
just test         # Run Rust tests, Python tests, and the maintained notebook
just test-rust    # Run only Rust tests
just test-python  # Run only Python tests
just marimo       # Open the notebook editor

Run the compiled event-to-vector throughput benchmark with:

cargo bench -p fiml --bench compiled_dispatch
# Run the workload assertions once without collecting timings:
cargo bench -p fiml --bench compiled_dispatch -- --test

It compares FeatureExtractor::handle_event and Pipeline::handle_event for 1, 4, and 16 subscribed symbols, with 1, 3, and 4 windows respectively (8–512 samples or time buckets). Benchmark IDs include raw/model output counts. The deterministic stream mixes price, volume, trade, and clock events, plus an unrelated symbol that still advances the global elapsed-time feature. Extractors compute timed SMAs and trade counts; pipelines add identity, standard scaling, sample SMA/EMA, and grouped event-lag transformations.

Construction, full-window warm-up, output assertions, and event batch creation are outside measurement. Each timed iteration drains a prebuilt batch into a persistent runtime, consumes outputs with black_box, and continues timestamps from the previous batch. Throughput counts all dispatched events, including unrelated symbols and clock ticks. Allocation guarantees remain covered by cargo test -p fiml --test pipeline_allocations; timings have no CI threshold.

Other docs

Build the Rust API reference from the repository root:

cargo doc -p fiml --all-features --no-deps --open

License

Fiml is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

About

Streaming financial indicators and feature pipelines for machine learning, with a shared Rust engine and Python bindings.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages