Skip to content
raws-labsPublic

About

Ahead-of-time compiler that fits ML models on embedded devices with hard memory budgets.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

7 stars

Watchers

0 watching

Forks

TiGrIS

License PyPI Docs

Tiled Graph Inference Scheduler. An ahead-of-time compiler that tiles ML models to fit embedded devices with hard memory budgets.

Give it a model and a memory budget: ONNX, TFLite, or a PyTorch program saved with torch.export (.pt2). It partitions the compute graph into stages, tiles spatial operations, and emits a flat binary plan that the tigris-runtime executes with zero dynamic allocation.

If you already deploy TFLite Micro or ExecuTorch, tigris export rewrites the int8 .tflite file or the Cortex-M .pte program in row bands instead, so the runtime you ship runs it in less RAM with the same outputs.

The problem

On an embedded device with a few hundred KB of SRAM, most interesting models simply don't fit. The usual answer is to shrink the model: quantize harder, prune, pick a smaller architecture, and hope the accuracy hit is acceptable.

TiGrIS takes the other approach. It keeps the model you trained and rearranges the computation so that only a small working set lives in SRAM at any moment. Weights can stay in flash, and intermediate results that do not fit spill to slower writable memory such as PSRAM. What comes out is a binary plan that the runtime executes as a flat sequence of kernel calls, with no interpreter: activations get their addresses from caller-provided fast and slow arenas by bounded bump allocation, and inference never calls a general-purpose allocator.

Installation

pip install tigris-ml selects a wheel for the host. Native wheels include the portable reference runtime for Linux x86-64 and ARM64 with glibc >= 2.28, macOS x86-64 and ARM64, and Windows x86-64. The Linux wheels use manylinux_2_28; older glibc, musllinux, Windows ARM, and 32-bit ARM Linux install the pure Python wheel instead. A source distribution is also available.

The pure wheel supports inspection, compilation, analysis, code generation, export of TFLite models and ExecuTorch programs, and zoo downloads. Running a plan uses the bundled native library; without it run reports "Bundled host runtime is unavailable". Running a TFLite model uses tflite-micro, and running an ExecuTorch program builds a host runner once from an ExecuTorch checkout, which fetches CMSIS-NN.

On platforms without a native wheel, build the host library from the matching tigris-runtime release: run cmake -S . -B build-host -DTIGRIS_BUILD_HOST=ON, then cmake --build build-host --target tigris_host, and set TIGRIS_HOST_LIBRARY to the resulting shared library file. tigris --version and tigris run --json report the runtime's version and origin; an invalid override fails without falling back.

Quick start

pip install tigris-ml

# Will this model fit in 256KB SRAM + 16MB flash?
tigris analyze mobilenetv2.onnx -m 256K -f 16M
warning: input axis 0 (batch_size) is unset; using 1 (--input-shape overrides)
mobilenetv2.onnx   float32, 65 operators
  input    input    1x3x224x224 float32
  output   output   1x1000 float32
fits 256.00 KiB fast memory, 58 stages, 53 tiled

memory
  unscheduled         5.74 MiB
  largest tensor      4.59 MiB   1x96x112x112
  this plan         250.00 KiB   6.00 KiB headroom
  slow memory         5.17 MiB
  slow traffic       20.25 MiB   per inference: 7.21 MiB written, 13.04 MiB read
  smallest budget    37.50 KiB   37.47 KiB does not fit

flash
  plan           13.31 MiB   weights 13.30 MiB, overhead 8.32 KiB
  as int8         3.33 MiB   estimate
  flash budget   16.00 MiB   fits

Unscheduled, the model peaks at 5.74 MiB. TiGrIS schedules it into 256 KiB through temporal partitioning and spatial tiling. analyze runs on your laptop; no hardware required.

That model is a stock export with a free batch dimension. TiGrIS binds a dimension the model leaves open to 1 and says so; pass --input-shape input:4x3x224x224 to compile for a different one.

From ONNX to embedded

Three steps take a model from ONNX to a C file you can drop into your firmware project:

# 1. Analyze feasibility against a memory budget
tigris analyze model.onnx -m 256K -f 16M

# 2. Compile to a binary plan (weights read-in-place from flash)
tigris compile model.onnx -m 256K -f 16M --xip -o model.tgrs

# 3. Generate a backend-specific C harness for your target
tigris codegen model.tgrs --backend esp-nn -o model.c

The .tgrs plan is target-agnostic: the same file runs on an ESP32, a Cortex-M, or a POSIX host. The kernel backend is chosen at codegen time and decides which kernel library the generated C calls into. The operator and backend matrix shows which operators are native, which fall back, and which are rejected.

What you get

tigris compile writes a single .tgrs file holding the operator schedule, tile parameters, quantization tables, and the weights. With a host runtime, bundled or named by TIGRIS_HOST_LIBRARY, it first loads the plan with the runtime's loader and writes no plan the loader refuses. The pure wheel without TIGRIS_HOST_LIBRARY writes the plan without that check.

tigris codegen produces a C harness that loads the plan and hands it to the runtime: buffer and arena declarations, a target entry point, and the glue for reaching the plan bytes. --format app emits a standalone program. --format core emits a source and header for firmware that already owns its entry point, arenas, and input source, so several generated cores can coexist in one binary. The codegen reference documents the flags.

Link the harness against tigris-runtime and your kernel library, and you have a working inference binary.

Precompiled models

The model zoo hosts precompiled plans with runtime requirements, input/output conventions, evaluation results, and per-model licenses. Public downloads need no account. List available builds before choosing a model:

tigris zoo list --category classification
tigris zoo list --runtime 0.9.1 --backend reference -m 256K
tigris zoo fetch MODEL -o downloaded-model
tigris inspect downloaded-model/model.tgrs
tigris codegen downloaded-model/model.tgrs --format core -o model.c

Filters are optional. The newest published matching build wins. Runtime ranges include both endpoints; a null maximum means no known upper compatibility bound. Tested releases are reported separately. Matching a range does not claim that every release in it has been tested. -m limits the fast arena, a second -m limits the slow arena, and -f limits plan bytes. These are not total application RAM or flash limits. Unspecified resources remain unconstrained. The runtime for deployment on a device is supplied separately.

Use fetch --artifact ID to pin a build. Withdrawn builds are excluded from automatic selection but remain explicitly retrievable with a warning. Downloads verify file hashes and never replace an existing destination. manifest.json preserves the original build metadata. download.json records the current runtime constraints, tested releases, and pinned catalog revision; use it for dependency integration. Catalog updates do not change artifact publication dates or rebuild models. tigris zoo --offline ... uses the HF cache; tigris zoo --catalog catalog.json ... reads a local zoo snapshot.

Inspect a model

tigris inspect model.onnx
tigris inspect model.tgrs -v
tigris inspect model.tgrs --json

inspect detects the format from file contents: ONNX, TFLite, an exported PyTorch program (.pt2), an ExecuTorch program (.pte), or a compiled plan. For a TFLite file it also states whether the model converts, and if not, what stops it; for an ExecuTorch program it shows the interface and the RAM the program records. It reads declared ONNX inputs, outputs, operators, and initializers, or compiled plan interfaces, stages, tiling, quantization, and memory records. -v adds details; --json emits complete metadata with an inspection_version field and exact byte counts, without weight values. Inspection is offline: it does not compile, execute, infer shapes, or load external ONNX tensor data. Recorded plan memory is not measured runtime peak or total application RAM. Plan schemas do not imply runtime version requirements. Inspection does not validate kernel numerics. run executes portable kernels on the host: TiGrIS's reference kernels for a plan, TFLite Micro's reference kernels for a TFLite model, CMSIS-NN's portable C kernels for an ExecuTorch program. ESP-NN, the DSP and MVE kernel paths and on-device latency run on the target.

Run a compiled model

Platform wheels include the reference C runtime. Run preprocessed tensors in the stored axis order and declared interface dtype reported by inspect:

tigris run downloaded-model/model.tgrs --input downloaded-model/example-input.bin --output prediction.bin

Compare prediction.bin with example-output.bin numerically; the artifact's readme.md names the reference (TFLite Micro or ONNX Runtime) and how close the outputs must be.

tigris run model.tgrs --input input.npy --output prediction.npy
tigris --version

For multiple inputs, repeat --input NAME=FILE. Use .npz for multiple named outputs. Raw .bin files contain contiguous little-endian elements; .npy inputs retain their declared shape and dtype. Output files must not exist. --json reports execution metadata and arena peaks. These peaks exclude the plan, executor workspace, and Python process memory. Host execution uses only the portable float32 reference and int8 reference kernels. It checks plan execution on that path, not ESP-NN or CMSIS-NN numerics or on-device latency. It does not preprocess photos or CSV files.

import numpy as np
from tigris.runtime import Session

with Session("model.tgrs") as session:
    outputs = session.run({"input": np.load("input.npy", allow_pickle=False)})
    print(session.runtime_version, session.memory)

Compiler and runtime releases share a version identifying the tested pair. Execution checks matching zoo dependency records when present and lets the C loader validate the plan. Schema versions, accepted schema ranges, and the host ABI retain their own compatibility meanings. Downloading and generating deployment code remain independent of the host runtime. See Contributing for native-library staging and wheel builds.

Further reading

Maintainer

TiGrIS is maintained by RAWS Labs. For applied embedded-ML engineering or collaboration, see raws.at.

Development

git clone https://github.com/raws-labs/tigris
cd tigris
pip install -e . --group dev
pytest

About

Ahead-of-time compiler that fits ML models on embedded devices with hard memory budgets.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

7 stars

Watchers

0 watching

Forks

Releases

Sponsor this project

Packages

Used by

Contributors

Languages