A deep learning framework for Lean 4 with a typed tensor facade over LibTorch.
Use torch.Tensor for shape, dtype, and device indices in model code.
Native adapters use RawTensor handles internally; checked constructors validate
external metadata before exposing indexed tensors.
import Tyr.Typed
open torch
def forward : Tensor { shape := #[32, 512], dtype := .Float32, device := .CPU } :=
let x := Tensor.ones #[32, 768]
let w := Tensor.ones #[512, 768]
x.linear w
-- Replacing w with Tensor.ones #[768, 512] is a type error.
-- Raw handles cross into this facade through Tensor.ofTensor, which checks
-- their actual shape and dtype. Native results enter through Tensor.ofRaw!.Install elan (the Lean version manager):
curl https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh -sSf | shOpen a new shell (or run source ~/.elan/env) so lake is on PATH. The
correct Lean nightly is pinned in lean-toolchain and will be installed
automatically on first lake build.
- macOS: Xcode command line tools (
xcode-select --install) - Linux: GCC 10+ (
sudo apt install build-essential)
Third-party libraries and sources are pinned in deps/ and fetched into
external/. This needs git, curl and unzip. Pick the CPU or CUDA build:
deps/fetch.sh cpu # or: deps/fetch.sh cuda# Configure once (Lake keeps it until the next `lake -R`); must match deps/fetch.sh:
lake -R build test_runner # CPU
# lake -R -Kcuda=/usr/local/cuda build test_runner # CUDA, with this toolkit
# Build the test runner (a good first build)
lake build test_runner
# Build specific executables
lake build TrainGPT
lake build TrainDiffusion
lake build TrainNanoChat
lake build FluxDemo
# Build everything (slow: every example and benchmark executable)
lake buildlake exe builds an executable if needed and runs it; arguments follow its name:
lake exe TrainGPTlake test # builds and runs test_runner
lake exe test_runner --filter GPT # with argumentsAll optional.
Dependencies (deps/fetch.sh):
| Variable | Effect |
|---|---|
TYR_DEPS_CACHE |
download cache directory (default external/.cache) |
Build configuration (Lake options, given together on lake -R; Lake keeps
them until the next lake -R, which replaces all of them):
| Option | Effect |
|---|---|
-Kcuda=<toolkit> |
CUDA build with this toolkit, which must contain bin/nvcc. Without it: CPU build (CUDA kernels replaced by stubs). Checked against the libtorch from deps/fetch.sh |
-Kgpu=<name> |
with -Kcuda, required: GPU to build kernels for, H100, A100, B200, B300 or GB10 |
Build:
| Variable | Effect |
|---|---|
TYR_MAKE_JOBS |
parallel jobs for the native make build (default: CPU count) |
See docs/ffi-and-build.md for finer GPU and compiler overrides.
Runtime:
| Variable | Effect |
|---|---|
TYR_DEVICE |
cpu, cuda, mps or auto; device used by the examples and model loaders |
TYR_VERBOSE_ERRORS=1 |
print the full libtorch report when a libtorch error crashes the program |
TYR_DEBUG_MPS=1 |
print MPS (Apple GPU) availability diagnostics |
Per-component guides live in docs/ — covering the core
tensor/FFI layer, training stack, model families, scientific computing, and
the C++/build internals. An exhaustive API reference can be
generated with doc-gen4 (see docbuild/).
See Examples/README.md for detailed per-example documentation.
| Example | Description | Build target |
|---|---|---|
| TrainGPT | Character-level GPT on Shakespeare | lake build TrainGPT |
| TrainDiffusion | Discrete masked diffusion on ASCII text | lake build TrainDiffusion |
| TrainNanoChat | Modded-nanogpt distributed training | lake build TrainNanoChat |
| FluxDemo | Flux Klein 4B image generation | lake build FluxDemo |
| NanoProof | Transformer theorem prover (model only) | Part of Examples lib |
Use the helper scripts to run TrainNanoChat under torchrun without pulling in a mismatched CUDA module stack:
# default: debug smoke run on 2 GPUs
./scripts/nanochat/run_train_torchrun.sh
# explicit 4-GPU run
NPROC_PER_NODE=4 ./scripts/nanochat/run_train_torchrun.sh \
--debug --iterations 2 --data data/nanochat --val data/nanochat
# scaling check (1/2/4 GPUs by default)
./scripts/nanochat/bench_distributed.shNotes:
- Override process counts in the benchmark script with
SIZES="2 4"(or any space-separated list).
Tensor σ tracks shape, dtype, and a concrete device in one static specification:
def project {m k n : Nat}
(x : Tensor { shape := #[m, k], dtype := .Float32, device := .CPU }) (w : Tensor { shape := #[k, n], dtype := .Float32, device := .CPU }) :
Tensor { shape := #[m, n], dtype := .Float32, device := .CPU } :=
x.mm wThe native carrier is RawTensor; model code uses indexed Tensor σ values.
Native results enter through Tensor.ofRaw!, which aborts if a result
contradicts its type; external data is checked on its own metadata where it
enters. Tensor.reshape requires equal element counts and treats #[]
as a scalar. Runtime-dependent shapes can be packaged existentially without
losing the tensor's shape index.
Generic traversal over structures containing tensors:
class TensorStruct (α : Type) where
map : (∀ {σ}, Tensor σ → Tensor σ) → α → α
mapM {m : Type → Type} [Monad m] : (∀ {σ}, Tensor σ → m (Tensor σ)) → α → m α
zipWith : (∀ {σ}, Tensor σ → Tensor σ → Tensor σ) → α → α → α
fold {β : Type} : (∀ {σ}, Tensor σ → β → β) → β → α → βUse Vector n α instead of Array α for type-safe zipWith operations.
Batch dimensions may come from runtime configuration while remaining in the tensor type:
let iter := BatchIterator.new shard 8 256
let (batch, iter') ← iter.next
-- batch : Option (Tensor { shape := #[iter.batchSize, iter.seqLen], dtype := .Int64, device := .CPU })Tyr includes a Lean command-level type provider for SafeTensors schemas:
import Tyr.SafeTensors
open torch
-- Works with a single .safetensors file or a sharded directory.
-- If `model.safetensors.index.json` exists, introspection follows `weight_map`.
safetensors_type_provider "/path/to/model_dir_or_file" as ModelWeights
def inspectWeights : IO Unit := do
IO.println s!"discovered tensors: {ModelWeights.tensorCount}"
-- Per-tensor typed loader + typed schema metadata
let tokEmbed ← ModelWeights.load_model_embed_tokens_weight
IO.println s!"embed dtype: {ModelWeights.model_embed_tokens_weightSpec.dtype}"
IO.println s!"embed shape: {tokEmbed.runtimeShape}"
-- Hierarchical aggregate generated from tensor names
let weights ← ModelWeights.loadAll
let qProj := weights.model.layers[0]!.self_attn.q_proj.weight
IO.println s!"q_proj shape: {qProj.runtimeShape}"
-- Hierarchical subtree loaders are also generated
let decoder ← ModelWeights.model.load
IO.println s!"loaded subtree"Notes:
- Tensor schema dtype metadata uses the core
torch.DTypetype (not raw strings). - For sharded checkpoints, unsafe index shard paths (absolute paths or
..traversal) are rejected.
The C++ bindings use careful reference counting. See cc/src/tyr.cpp header for details:
borrowTensor(): Shared ownership, auto-cleanupgiveTensor(): Transfer ownership to Leanlean_dec(): Required after extracting fromlean_obj_arg, not forb_lean_obj_arg
Monitor tensor leaks via get_live_tensors which tracks outstanding C++ tensors.
- Add Lean declaration in
Tyr/RawTensor.leanor aTyr/RawTensor/submodule with@[extern "lean_torch_xxx"] - Implement in
cc/src/tyr.cppfollowing reference counting conventions - Rebuild:
lake build
lakefile.lean- Lake build configurationlean-toolchain- Lean version specificationcc/- C++ FFI bindings (LibTorch wrapper)Tyr/- Core framework (tensors, modules, optimizers, distributed)Examples/- Training scripts and model implementationsTests/- Test suites
This repo uses scoped conventional commit subjects:
type(scope): summary
A commit message template is included at .gitmessage. Enable it and the
hooks locally:
git config commit.template .gitmessage
git config core.hooksPath .githooksIncluded hooks:
pre-commit: fails on staged whitespace errors and conflict markerscommit-msg: enforcestype(scope): summary(e.g.feat(qwen35): add video stream patchify)pre-push: validates pushed commit subjects with.githooks/check-commit-message.sh
CI also enforces this format on pull requests, for both commit subjects and the PR title (a squash merge uses it as the commit subject), using the same checker.
Manual check example:
./.githooks/check-commit-message.sh "feat(qwen35): add video stream patchify"Licensed under the Apache License, Version 2.0.