Run the same PyTorch model on any supported accelerator by changing one string.

LM7 is a PyTorch-first compiler orchestration layer for local inference. Hand it a model you already run — a pretrained network, a Hugging Face causal LM, a vision model, a single layer — and get back a normal callable.

import lm7
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct").eval()

model = lm7.compile(model, target="auto")  # detect the best local device
model = lm7.compile(model, target="nvidia:sm89")  # or pin it exactly

lm7.compile returns a normal nn.Module, so the rest of your code does not change — no device juggling, no per-vendor branches, no manual compile cache. LM7 resolves the target, picks a compatible compiler, moves the inputs, compiles once per input shape, and falls back to plain PyTorch if a backend cannot handle the model.

LM7 writes no kernels and no compiler of its own. Every vendor already ships one; LM7 drives it. The same call site reaches a different vendor toolchain based only on the target you name:

lm7.compile(model, target="cpu")  # TorchInductor, C++/OpenMP
lm7.compile(model, target="nvidia")  # TorchInductor, Triton + cuBLAS/cuDNN
lm7.compile(model, target="nvidia", backend="tensorrt")  # Torch-TensorRT instead
lm7.compile(model, target="apple")  # TorchInductor, Metal via MPS
lm7.compile(model, target="tpu")  # PyTorch/XLA and OpenXLA
lm7.compile(model, target="tenstorrent")  # tt-xla, tt-mlir, tt-metal
lm7.compile(model, target="intel:npu")  # OpenVINO, Intel NPU plugin

You do not install or learn five toolchains to try a second device; you change a string, and lm7 doctor reports what is missing. The corollary is that LM7's reach is bounded by what those toolchains already support.

For the per-vendor code this replaces — detection branches, device-string inconsistencies, and the behaviour you would otherwise have to know about — see what LM7 replaces.

• Target and backend are separate. A target is where the model runs (cpu, nvidia, apple, tpu, …); a backend is the compiler used to get there (inductor, tensorrt, openxla, …). Pin either, or let LM7 choose. That split is what makes hardware swappable.
• Detection is automatic. target="auto" prefers a detected GPU or accelerator, and otherwise uses the CPU.
• JIT compilation is lazy and per-input-shape. Nothing compiles until the first call; that call is slow and later calls are fast. AOT moves that cost out of the process — see JIT vs. AOT.
• Fallback is safe by default. If a backend fails to compile, LM7 falls back to PyTorch eager and warns. Use fallback="error" to stop instead.

Any x86-64 or ARM64 CPU runs through cpu, Intel and AMD included. A vendor listed more than once has additional accelerators; its CPU is still supported. Backends in parentheses are export-only or explicit — see backends.

Add a qualifier to pin an architecture, model, or ordinal — nvidia:sm89, amd:gfx942, cpu:arm64, tenstorrent:blackhole.

intel:npu is the one target with no PyTorch device behind it: OpenVINO owns the NPU, so inductor and eager decline it, target="auto" never picks it, and it has its own guide.

Run lm7 targets to see what is actually present on your machine.

Not supported, with written plans: Qualcomm Hexagon NPU and AMD MIGraphX.

Smoke-test coverage, not a model zoo — LM7 doesn't allowlist models, so other Hugging Face checkpoints will likely run, just without this validation. See limitations and quantization for details.

LM7 needs Python 3.10+ and a PyTorch build matching the target machine. It does not install GPU drivers, CUDA/ROCm toolchains, Xcode, PyTorch/XLA, or C++ compilers.

git clone https://github.com/lmontigny/lm7.git
cd lm7
uv venv --python 3.12
uv pip install -e .            # add ".[dev]" for pytest + ruff

uv also picks the right PyTorch wheel, which matters more than usual here — CPU, CUDA, and ROCm builds come from different indexes:

uv pip install torch --torch-backend=auto

Then activate the environment (source .venv/bin/activate) or prefix commands with uv run. Without uv: python3 -m venv .venv && source .venv/bin/activate && python -m pip install -e ..

Per-hardware setup: CPU · NVIDIA · AMD ROCm · Apple Silicon · Google TPU · Tenstorrent.

lm7 doctor                    # environment and install check
lm7 targets                   # detected hardware targets
lm7 backends                  # registered compiler backends
lm7 explain --target auto     # which backend LM7 would pick, and why

Add --json to any command for machine-readable output. The same CLI is available as python -m lm7.

target="auto" selects hardware for you; the first real call compiles that input signature.

compiled = lm7.compile(model.eval(), target="auto")
result = compiled(example_input)

print(compiled.target, compiled.selected_backend)

With backend="auto", LM7 picks the highest-priority backend that supports the resolved target: inductor on CPU, NVIDIA, AMD, Intel, and Apple; openxla on TPU; tenstorrent on Tenstorrent; openvino on the Intel NPU, where it is the only candidate. eager wins only when nothing else supports the target, or when a compile fails and fallback="warn" takes over.

Two rows in that table are not lm7.compile backends at all. Export only means the backend is reachable solely through lm7.export(..., backend=...) — asking lm7.compile for one raises. Explicit means it works with lm7.compile but is never chosen by backend="auto"; tvm is the only one, because its untuned codegen is far slower than Inductor (details).

On NVIDIA, inductor is the default and tensorrt the opt-in alternative, deliberately lower priority because TensorRT builds engines more slowly and covers fewer models. On a local RTX 4070 SUPER, TensorRT beat Inductor 1.76x on a fixed-shape SmolLM2 FP16 forward pass, lost on two small MLPs, and took 56 seconds for the SmolLM2 first call — see the evaluation. LM7 never invokes Triton itself; TorchInductor owns kernel generation and selection.

Most non-default backends need an extra:

uv pip install -e ".[tensorrt]"      # Torch-TensorRT 2.12.1 / PyTorch 2.12 / CUDA 13
uv pip install -e ".[openvino]"      # Intel CPU and NPU
uv pip install -e ".[onnxruntime]"   # CPU — or ".[onnxruntime-gpu]" for CUDA 13, never both
uv pip install -e ".[iree-vulkan]"   # Vulkan AOT export
uv pip install -e ".[openxla]"       # on a TPU VM
uv pip install -e ".[tvm]"           # Apache TVM

Three need their own environment, because they pin PyTorch or ship outside PyPI:

uv pip install -e ".[litert]"        # LiteRT Torch caps PyTorch below 2.13
uv pip install -e ".[stablehlo]"     # torch_xla, ABI-tied to a matching PyTorch
uv pip install -e ".[executorch]"    # runtime extension ABI-linked to libtorch
uv pip install pjrt-plugin-tt --extra-index-url https://pypi.eng.aws.tenstorrent.com/

LM7_TARGET, LM7_BACKEND, LM7_FALLBACK, and LM7_CACHE_DIR set defaults; explicit arguments take precedence.

The difference is when compilation happens and whether the result outlives the process.

• JIT compiles inside your process on the first call, once per input signature. Nothing is written that another process can use, so restarting recompiles.
• AOT (lm7.export) compiles up front and writes an .lm7 directory another process loads with no compile step.

Use JIT while iterating, AOT when you want the compile cost paid once at build time. See JIT vs. AOT for the export levels, bundles, and the signature rules an artifact is pinned to.

uv pip install -e ".[hf]"
lm7 model run hf://HuggingFaceTB/SmolLM2-135M-Instruct \
  --prompt "The capital of France is" --target auto --backend auto

The command downloads through the normal Hugging Face cache, compiles the forward pass, and reports the selected target and backend, first-call and steady-call time, and the predicted next token (--json for structured output).

Validated compact, ungated models: HuggingFaceTB/SmolLM2-135M-Instruct, LiquidAI/LFM2.5-230M, unsloth/Llama-3.2-1B-Instruct, Qwen/Qwen3.5-0.8B (hybrid architecture; noticeably slower first call), and deepseek-ai/deepseek-coder-1.3b-instruct. The first four also compile on Apple Silicon, with slightly wider float16 tolerance on MPS than CUDA; DeepSeek was validated on CPU and NVIDIA only, across every backend installable on one host — see DeepSeek coverage.

For greedy token generation, use the static KV-cache path:

lm7 model generate hf://HuggingFaceTB/SmolLM2-135M-Instruct \
  --prompt "The capital of France is" --max-new-tokens 32 --target nvidia

Generation runs the prefill eagerly, then reuses one Inductor-compiled, fixed-shape decode graph against a static KV cache for every token. Greedy-only, and requires a Transformers causal LM supporting static caching — see compiled generation.

Capture a model and reload it in another process:

artifact = lm7.export(model, args=(example_input,), target="cpu", output="model.lm7")

loaded = lm7.load_artifact("model.lm7")
output = loaded(example_input)

An .lm7 artifact is a directory with a versioned manifest, checksums, and a PyTorch .pt2 program. Pick a backend to add a compiled payload beside it:

Two payloads are not bound to the machine that built them: stablehlo, whose plugin is chosen at load time, and executorch, whose XNNPACK delegate covers ARM64 and x86-64 alike. Everything else is specific to compatible compiler, runtime, and hardware versions — artifacts are not a stable cross-version ABI.

Export a Hugging Face model without writing any PyTorch:

lm7 model export hf://HuggingFaceTB/SmolLM2-135M-Instruct model.lm7 \
  --target cpu --backend aot_inductor

Example inputs come from tokenizing --prompt, so the artifact is pinned to that many tokens by default. --dynamic-seq captures the sequence length as a bounded dynamic dimension instead:

lm7 model export hf://HuggingFaceTB/SmolLM2-135M-Instruct model.lm7 \
  --target nvidia --backend aot_inductor --dynamic-seq        # bounds from the config
lm7 model export hf://HuggingFaceTB/SmolLM2-135M-Instruct model.lm7 \
  --target nvidia --backend aot_inductor --dynamic-seq 1:512  # or set them

Bounds are recorded in the manifest and enforced on every call, so a prompt outside them raises rather than producing quietly wrong output.

Combine per-target artifacts and select at load time:

lm7.create_bundle(["build/cpu.lm7", "build/nvidia.lm7"], output="model.bundle.lm7")
deployed = lm7.load_bundle("model.bundle.lm7").load(target="auto")

lm7 bundle create model.bundle.lm7 build/cpu.lm7 build/nvidia.lm7
lm7 bundle inspect model.bundle.lm7      # add --json for structured output

For lm7 model run, quantization stores weights in fewer bits than the model was trained in. This path is weight-only:

uv pip install -e ".[hf,torchao]"
lm7 model run hf://HuggingFaceTB/SmolLM2-135M-Instruct \
  --target cpu --quantize int8      # 513 -> 210 MiB, same next token

The conversion is TorchAO's. It is opt-in and admitted per (model, mode) pair — a mode is rejected for a model whose outputs have not been compared against an unquantized baseline on real hardware. int8 is the only mode measured off NVIDIA; on CPU it cuts a model to 2.44x smaller at no measured accuracy cost, with a latency effect that depends on model size. nvfp4 buys the smallest footprint and costs the most accuracy; it clears the bar for one model out of four tried. Both --quantize and the older --quantization spelling work.

This runtime path does not quantize activations. Two export paths quantize the artifact instead, each with its own mechanism:

lm7 model export hf://... out.lm7 --backend openvino  --quantize int8  # NNCF, IR weights
lm7 model export hf://... out.lm7 --backend executorch --quantize int8  # calibrated XNNPACK PTQ

The OpenVINO one compresses the IR to 3.98x smaller and was the only quantization measured here that runs faster than its FP32 baseline. See quantization for every mode and the measurements behind them, and ExecuTorch for the edge flow.

python examples/basic_mlp.py                        # CPU
python examples/cuda_mlp.py --target nvidia         # NVIDIA
python examples/mac_mlp.py                          # Apple Silicon
python examples/tenstorrent_mlp.py                  # Tenstorrent
python examples/local_targets.py --require-nvidia   # CPU vs NVIDIA parity
python benchmarks/local.py --target cpu nvidia --backend eager inductor

More in examples/ and benchmarks/.

The documentation index lists everything. Most useful:

• Limitations — what LM7 does not do, per backend.
• Architecture — targets, backends, planner, artifacts.
• JIT vs. AOT — compilation timing and artifact rules.
• Development and testing — running the suite, GPU tests.

Issues and pull requests are welcome. Run the checks before opening one:

uv pip install -e ".[dev]"
python -m pytest -q
python -m ruff check .
python -m ruff format --check .

Hardware-specific tests skip automatically when the device or toolchain is absent. See development and testing.

LM7 is licensed under the BSD 3-Clause License.