Skip to content
jallumPublic

About

No description, website, or topics provided.

Resources

Stars

8 stars

Watchers

0 watching

Forks

Repository files navigation

CI coverage

fz - a friendly, functional language

The BEAM got some things profoundly right: cheap processes, isolated heaps, message-passing, pattern matching at the core, selective receive. Forty years on, that's still the model for fault-tolerant concurrent systems, and nobody has improved on the semantics.

The implementation strategy was shaped by what was possible in an interpreter built in the 80s. fz keeps the semantics and asks what a whole-program compiler with types can do with them.

What that means in practice: fz reads like Elixir, runs the actor model you already know, and compiles — through a set-theoretic type system, a Cranelift JIT, and an AOT path — to native code. One IR powers three execution modes (AOT executable, JIT, interpreter) -- and soon a REPL, and a fixture matrix forces them to agree.

def add(a, b), do: a + b

def main() do
  dbg(add(2, 3))
end

A file can also be a script: its top-level expressions run in order as the program's main/0 (a file with both top-level expressions and its own def main() is an error).

Having full control of the stack buys some flexibility in interesting places: When you write receive in fz, the receiver's pattern is compiled into a matcher that can also run on the sender's heap, and only the values the receiver actually bound cross the process boundary. Everything else stays on the cutting room floor. Same Elixir-shaped program, same semantics; the machine does less work.

The same control buys another trick we think you'll like: the compiler builds values directly into their final position and reuses list cells it can prove are private — so textbook functional code runs with the allocation profile of a hand-tuned loop, and you never write a single borrow annotation to get it.

If you're an Elixir or Erlang person curious about what a typed, natively-compiled cousin looks like — or a compiler person who wants to see the internals of a language that takes the BEAM seriously — keep reading.


The bet

Three deliberate choices.

1. Elixir's surface, because it's pleasant (thank you, José!)

If you've written Elixir, fz will read like a slightly stripped-down dialect in some ways, more fleshed out in others. Same def name(args), do: body, same case / with / receive, same atoms, tuples, lists, maps, binaries, same defmodule, same @type / @spec, same defmacro + quote / unquote, same |>. We borrowed the surface syntax wholesale because Elixir is one of the most pleasant functional languages to read, and there was no reason to relearn that lesson.

2. BEAM's concurrency model, because it works

The BEAM has spent forty years getting a handful of ideas extremely right, and we are taking those wholesale:

  • isolated processes, each with its own private heap, so "no shared mutable state" is enforced by the runtime instead of by convention
  • message passing as the only way processes interact — copy on send, no aliasing across the boundary
  • pattern matching at the core of the language, not bolted on the side — function clauses, case, with, and receive all run through the same matcher
  • lightweight processes by the thousand, cooperatively scheduled at receive and compiler-inserted yield points, cheap to spawn and cheap to discard
  • selective receive, so a process can wait for the exact message it cares about instead of inventing a state machine around next()

We are not trying to reinvent the wheel -- wheels are great. We want a faster one that can ship native binaries.

The long-term goal is full interop: an fz node speaking Erlang's external term format and distribution protocol, joining an existing BEAM cluster as just another participant in the supervision tree. Not "migrate to fz" — "add an fz node to the cluster." That work is sequenced behind getting the local semantics right first.

3. A real compiler underneath, because that's where the win lives

Elixir compiles to BEAM bytecode. fz has its own compiler and its own native runtime, written in Rust: an interpreter, a Cranelift-based JIT, an AOT path that produces real executables, and a REPL — all sharing one IR.

The compiler doesn't exist for its own sake. It exists so we can do things the BEAM's interpreter structurally couldn't — and the two tricks teased above are both deliberate payoffs of owning the whole stack, from source to machine code:

  • Running a receiver's matcher on the sender's heap, so only the values a receive actually asked for ever cross the process boundary. (Selective receive, sharpened, below.)
  • Building values straight into their final position, so textbook functional code allocates almost nothing — no reference counting, no annotations. (All of it together: quicksort, below.)

Both keep BEAM semantics exactly. The compiler just moves work ahead of time so the machine does less of it at runtime.


A guided tour

Pattern matching is how you make decisions

A function can have several clauses. fz picks the first one whose shape matches:

def length([]), do: 0
def length([_ | rest]), do: 1 + length(rest)

def describe(0), do: :zero
def describe(1), do: :one
def describe(_), do: :many

Under the hood, every match in the language — function clauses, case, with, even receive — feeds into one tiny "sorting machine": ask yes/no questions about the shape, pluck out the pieces that matter, hand the winning branch the values it asked for. One machine, four surface forms. The machine also destructures shared shape exactly once — when two clauses both match [h | t], as partition does in the quicksort below, the cons cell gets walked one time across both clauses, not twice. (pattern-matching guide)

Values are immutable

Lists, tuples, maps, binaries, atoms, integers, floats, UTF-8 strings — all values. You don't mutate them; you make new ones:

def swap({a, b}), do: {b, a}

def main() do
  dbg(swap({:left, :right}))   # {:right, :left}
end

This is the load-bearing rule that makes the concurrency story safe: if nobody can mutate anything, nobody can race over anything.

First-class functions, closures, simple macros

Functions are values: pass them around, return them, capture their environment in a closure. Macros run at compile time, rewriting code before it is lowered.

def double(x), do: x * 2
def compose(f, g, x), do: f.(g.(x))

defmacro inc(x) do
  quote do: unquote(x) + 1
end

def add1(x), do: inc(x)

def main() do
  dbg(compose(double, add1, 20))   # 42
end

Those closures are what the concurrency examples below spawn — when you write spawn(fn() -> relay(n, home) end), that anonymous function is a first-class value handed to a brand-new process.

Concurrency: processes, not threads

A process is a tiny unit of execution with its own stack, its own heap, and its own mailbox. You spin up thousands; they don't share memory; they talk by passing messages.

Three primitives carry the entire story:

  • spawn(f) — start a new process running f, return its pid
  • send(pid, msg) — drop a message in someone's mailbox
  • receive do … end — pattern-match a message out of your own mailbox

Smallest possible ping:

def child(), do: send(1, 42)

def main() do
  spawn(child)
  dbg(receive do x -> x end)   # 42
end

A ring of processes, each adding 1 and passing the value on:

def relay(0, home) do
  value = receive do x -> x end
  send(home, value + 1)
end

def relay(n, home) do
  next = spawn(fn() -> relay(n - 1, home) end)
  value = receive do x -> x end
  send(next, value + 1)
end

def main() do
  home = self()
  head = spawn(fn() -> relay(4, home))
  send(head, 0)
  got = receive do x -> x end
  dbg(got)   # 5
end

When a value is sent across processes, the sender measures it and copies it once, into one block of memory the recipient then adopts. There are no shared references because there are no references — only values. A process that sends to itself copies nothing: the message goes straight onto its own mailbox.

Selective receive, sharpened

This one is pretty. Here's an ordinary-looking program — a tiny server that echoes back a key, and a client that asks it two questions:

def handle_get(ref, from, key) do
  send(from, {:reply, ref, key})
  server()
end

def server() do
  receive do
    {:get, ref, from, key} -> handle_get(ref, from, key)
    {:stop}                -> nil
  end
end

def main() do
  s     = spawn(server)
  ref_a = make_ref()
  ref_b = make_ref()
  me    = self()

  send(s, {:get, ref_a, me, 1})
  send(s, {:get, ref_b, me, 2})

  # Replies arrive in ref_a, ref_b order, but we want ref_b's first.
  val_b = receive do
    {:reply, ^ref_b, v} -> v
  after
    500 -> :timeout
  end

  # ref_a's reply has been sitting in the mailbox the whole time.
  val_a = receive do
    {:reply, ^ref_a, v} -> v
  end

  dbg(val_a + val_b)
  send(s, {:stop})
end

The Elixir programmer reads this and thinks: "yes, that's selective receive — wait for the message I actually want, leave the rest in the mailbox for later." The behaviour is identical to BEAM's. What's different is how the runtime gets there.

When the receiver writes receive do {:reply, ^ref_b, v} -> v end, the compiler lowers that pattern into a tiny matcher program: a constant-time decision tree that knows exactly which shapes the receiver will accept and which pieces of those shapes (v) it wants to bind. The compiler also wraps the winning clause's body and its captured bindings into a continuation — a closure ready to run. This matcher can be used to scan the mailbox, but it can also be run on the sending side.

So when send(...) runs:

  • If the message matches, the sender copies only the values the pattern bound into one fresh block of its own memory and queues that block for the receiver. The parts of the message the receiver didn't name (the :reply tag, the matched ref, anything else) stay in the sender's heap and get collected normally. The receiver wakes up, adopts the segment, builds the resumption closure over the values it asked for, and the clause body runs — no rescan, no rebinding, no branch selection on the receiver side. A pattern that binds nothing copies nothing.
  • If the message doesn't match the parked receiver, BEAM semantics still apply: the message gets enqueued at the end of the mailbox (a later receive might want it), and the parked receiver stays parked. What we don't do is wake the receiver up just to look at a message we already know it would reject. The sender copies the whole message once, into one block the receiver adopts the next time it runs.

That's the trick: compilation lets us turn "send the whole message, then match it on arrival" into "match first, then send only what was asked for" — while keeping BEAM mailbox semantics intact. The receiver told us, by writing a receive, exactly what it cared about; the sender does the work on its own time and its own heap.

A message is measured, then copied once by the sender into one block of memory the receiver adopts. No process ever writes into another process's heap: the receiver grafts the sender's block onto its own heap with pointer moves, and its next garbage collection copies out whatever is still live. Because the receiver's matcher runs on the sender, a when guard in a receive runs there too, so it must be pure; a call such as self() in a guard names the receiver, because the receiver evaluates it before it parks.

The receiver gets concierge treatment.

All of it together: quicksort

def append([], ys), do: ys
def append([h | t], ys), do: [h | append(t, ys)]

def partition(_, [], lo, hi), do: {lo, hi}
def partition(p, [h | t], lo, hi) when h < p, do: partition(p, t, [h | lo], hi)
def partition(p, [h | t], lo, hi), do: partition(p, t, lo, [h | hi])

def qsort([]), do: []
def qsort([p | rest]) do
  {lo, hi} = partition(p, rest, [], [])
  append(qsort(lo), [p | qsort(hi)])
end

def main() do
  dbg(qsort([3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5]))
end

Pattern matching, recursion, tuples, lists, multiple clauses — that's the whole sort, and it's readable end to end. It's also where the second trick from The bet pays off.

The folklore says elegance like this costs allocations: a {lo, hi} tuple built only to be torn apart, the whole lo list built and then walked again and re-copied by append, a closure for every "what to do when this returns." fz mostly doesn't pay. Two compiler ideas get us there, and neither asks anything of you:

  • Destination planning — before codegen, the planner asks each call not just what type it returns but where the result is going. If the caller is about to destructure a pair, the callee hands back two register values and never builds the tuple. If a list's eventual tail is already known, the callee builds its cons cells pointing at that tail from the start — so the append that would walk and re-copy the list has nothing left to do.
  • Owned-cons reuse — when the compiler can prove a cons cell is still private, the runtime reuses it in place instead of allocating. A single alias bit in the cell guards it: still private, relink it; already published, allocate a fresh one. This doesn't bend the immutability rule from above — the compiler only exploits privacy the program can never observe.

The payoff is measurable and pinned. This exact program, sorting its eleven-element list, allocates exactly the eleven input cons cells — zero tuples, zero continuation closures on the native paths:

list_cons_allocs = 11     # the input list's cells, nothing more
struct_allocs    = 0      # no {lo, hi} tuple ever hits the heap
closure_allocs   = 0      # no continuation closures on the fast path

Here's what we like: no reference counting, no borrowing annotations. Koka-style reuse leans on runtime refcounts; Rust-style reuse leans on lifetimes and & in your source. fz does neither — the source is the same direct functional code, with no unique, no linearity, no "this list is mine." The compiler proves it statically; the runtime pays one bit. You get the readability of pure functional code with the memory profile of a hand-tuned loop.

This is the JIT and the AOT compiler doing their job — turning the program you'd actually ship into native code that allocates like you hand-tuned it. (The interpreter and REPL run the same code to the same answers; they're there to keep us honest, not to win the allocation game.) (the full walkthrough)

The same machinery now applies through the standard library too. The runtime-library Enum.sort/1 is a merge sort with a default comparator closure. Native code proves that comparator is constant, erases it from the recursive sort frames, and keeps the hot path closure-free:

Enum.sort list_cons_allocs = 22
Enum.sort closure_allocs   = 0
Enum.sort heap_bytes       = 352

That is still a general-purpose library sort rather than hand-shaped quicksort, so it allocates more cons cells. The important part is that the abstraction boundary is no longer where optimization stops.

Talking to C

The OS lives in C. fz doesn't try to hide that — it declares it:

extern "C" def libc::creat(path :: c_string, mode :: integer) :: c_int
extern "C" def libc::write(c_int, binary, integer) :: integer
extern "C" def libc::close(c_int) :: c_int

def main() do
  fd = libc::creat("/tmp/hello", 420)   # 0o644
  libc::write(fd, <<104, 105, 10>>, 3)  # "hi\n"
  libc::close(fd)
end

Each argument has a marshal class (integer, c_int, binary, c_string, any, nil) telling fz how to translate between tagged fz values and the slots C expects. c_int is the one that says a value is 32 bits wide, so an error sentinel like -1 stays -1 on a machine whose C library leaves the other half of the register alone. The c_string class is a small trick: every fz binary has an invisible trailing zero byte past its end, so handing a binary to a C function that wants a char * is a pointer pass, not a copy. You can also attach a destructor to a resource and let fz clean up after the C side when the value goes away. (externs guide)

The point: the standard library — file I/O, time, networking, whatever — can be built out of fz code that calls C, instead of waiting on the compiler team to add features one by one.


Under the hood

Types do two jobs

Types catch mistakes — wrong shapes, missing clauses, the things you'd otherwise find at runtime. They also let the compiler skip work: the more fz can prove about a value, the more direct the code it can emit. You get both from the same investment.

def main() do
  x = 41 + 1
  dbg(x)          # compiler knows x is an integer; uses the int debug path
end
def kind(0), do: :zero
def kind(n) when n > 0, do: :positive
def kind(_), do: :other

def main() do
  dbg(kind(5))   # compiler proves which clause wins, drops the rest
end

The type system is set-theoretic — unions, intersections, negations — following the Castagna line that Elixir's own typer is built on. Keep the source language small and pleasant; teach the compiler to learn as much as it can from it.

The pipeline

source code
  -> compiler2 source publication
  -> local semantic facts
  -> product-keyed artifact handoff
  -> run it: interpreter, JIT, or AOT

One compiler2 source and semantic fact model, product-keyed artifacts for the runtime handoff. The ways to run it must agree at the observable boundary.

The type stack is split by ownership. src/compiler2/types/ owns compiler2's interned set-theoretic lattice, src/compiler2/type_expr.rs turns source type syntax into compiler2 facts, and compiler2 derives ABI, backend, and native artifacts from those facts by named demand.

Looking inside the compiler

The rule in this repo: do not guess. Make the compiler leave breadcrumbs.

fz2 run --dump clif=/tmp/quicksort.clif fixtures2/behavior/quicksort.fz
fz2 run --dump types=/tmp/quicksort.types fixtures2/behavior/quicksort.fz
fz2 run --dump activations=/tmp/quicksort.activations fixtures2/behavior/quicksort.fz
fz2 run --dump backend=/tmp/quicksort.backend fixtures2/behavior/quicksort.fz
fz2 run --dump native=/tmp/quicksort.native fixtures2/behavior/quicksort.fz

Compiler2 dumps are file sinks installed on a run, interp, or build command. The supported dump kinds are activations, types, backend, native, fnir, and clif.

Run/build/interp commands always submit the root source directly to compiler2. If the program pulls in built-in runtime modules such as Utf8 or Enum, the compiler loads their checked-in source on demand and materializes only the root-reachable artifact set.

These answer the questions you actually have while changing things: Did this call get folded? Did this function get specialized? Did the compiler skip something — and why? Compiler2 telemetry and fixture contracts are the durable signal for compiler shape.


Trying it

Build the compiler:

cargo build --release

That gives you a fz2 binary at target/release/fz2 — put it on your PATH (or alias it) and the rest of these commands just work.

Run a file with the JIT:

fz2 run fixtures2/behavior/quicksort.fz

fz2's options go before the source path. Every token after the path belongs to the program, verbatim, and System.argv() returns them as a list of binaries (-- is an ordinary argument):

fz2 run script.fz input.txt --verbose    # System.argv() == ["input.txt", "--verbose"]

A built executable reads its own command line the same way:

fz2 build script.fz -o /tmp/script
/tmp/script input.txt --verbose

Build a native executable:

fz2 build fixtures2/behavior/quicksort.fz -o /tmp/qsort
/tmp/qsort

Run through the interpreter:

fz2 interp fixtures2/behavior/quicksort.fz

Start the REPL:

fz repl

Run the whole test suite:

cargo test --workspace

The repository's Cargo configuration defaults to compact output: passing tests print progress characters instead of individual names, followed by each test binary's summary. Failure diagnostics and compiler warnings remain visible. Existing test commands work unchanged, including fixture tests and doctests. Use cargo --config term.quiet=false test --workspace -- --format pretty to restore individual test names (the custom fixture harness also defaults to terse); -- --nocapture still exposes test output when investigating a test.

To suppress passing-test progress too, use cargo-nextest and run doctests separately:

cargo nextest run --workspace --show-progress none --color never
cargo test --workspace --doc

The repository's nextest defaults suppress individual passing results and successful-test output. Failed tests print their captured diagnostics immediately, once, and the final summary lists failures. Avoid --nocapture when keeping output compact. Add --no-fail-fast to finish the suite after a failure.

Fixture tests run small .fz programs and compare their output across every execution path that applies (JIT, interpreter, AOT executable, REPL script mode). A fixture is more than a sample file — it is a tiny promise about the language. If quicksort works in the JIT but not in AOT, the fixture matrix catches it.

Fixture conventions — how each fixture pins its claim — are explained in .agent/docs/fixtures.md.

CI runs lint, security audit, doctests, and workspace tests for pull requests and pushes to main and develop. Pushes to either branch also enforce the coverage threshold; only main updates the published coverage badge.

Pre-commit hook

A tracked pre-commit hook lives in .githooks/. It formats the staged Rust with cargo fmt --all, re-stages it, then runs cargo check --all and cargo clippy --workspace --all-targets -- -D warnings. The lint set itself lives in the workspace manifest, so the hook and CI both enforce the same ownership/cloning policy without carrying a second copy of the lint list. A clippy complaint aborts the commit before it ever reaches CI. Git looks for hooks in .git/hooks by default, so enable the tracked directory once per clone:

git config core.hooksPath .githooks

What's in the box today

  • integers, floats, atoms, booleans, nil, tuples, lists, maps, binaries, UTF-8 strings
  • immutable values
  • set-theoretic types with @type and @spec declarations
  • pattern matching (function clauses, case, with, receive)
  • multi-clause functions and guards
  • modules, imports, simple macros
  • first-class functions and closures
  • processes: spawn, self, send, receive, refs, selective receive with sender-side matching
  • a working interpreter, JIT (Cranelift), AOT path, and REPL
  • C externs with marshal classes and resource destructors
  • destination planning + reusable-cons transport: textbook functional code compiled to near-zero-allocation native code (JIT/AOT), no reference counting, no borrow annotations

Status

fz is early. The compiler, the runtime, four execution paths, the type system, the sender-side matcher, and the destination-planning / reusable-cons path that gives native code its near-zero-allocation profile are all working today — but the language is small, the standard library is smaller, and the edges are sharp. Expect to read the dumps when things surprise you.

What's next, roughly in order:

  • OTP behaviors (gen_server, supervisor, links, monitors) — built in fz on top of the existing process primitives
  • Blocking-aware externs so libc::read doesn't stall a scheduler thread
  • Distribution via ETF and disterl — fz nodes that join existing BEAM clusters as ordinary participants in the supervision tree
  • An FBIP tensor type — the same in-place machinery, extended to Nx-shaped buffers: Nx-shape ergonomics, in-place buffer reuse, no defn ceremony, BLAS/MKL via externs
  • Autodiff over fz IR

The current focus is keeping the four execution paths in lockstep and teaching the compiler to turn more obvious functional code into efficient native code. Every other goal sits on top of that one.

Editor Support

The vscode-fz/ folder contains a minimal VS Code extension for FZ syntax highlighting and editor defaults. It is TextMate-based only for now: no language server, no diagnostics, just file association and a grammar tuned to the language surface.

Papers that shaped fz

fz didn't invent its good ideas — it borrowed them from people who worked them out properly. The ones that left the deepest marks:

  • Luc Maranget, Compiling Pattern Matching to Good Decision Trees (ML '08). The "one tiny sorting machine" behind every match in the language — the constant-time decision tree that asks yes/no questions about shape and destructures shared structure exactly once — is this paper. pdf

  • Alex Reinking, Ningning Xie, Leonardo de Moura & Daan Leijen, Perceus: Garbage Free Reference Counting with Reuse (PLDI '21). The "garbage-free, reuse cells you can prove are private" goal behind reusable-cons transport. Perceus reaches it with runtime reference counts; fz chases the same near-zero-allocation profile but proves privacy statically and pays a single alias bit instead. pdf

  • Anton Lorenzen, Daan Leijen & Wouter Swierstra, FP²: Fully in-Place Functional Programming (ICFP '23). The discipline that says when ordinary functional code can run with no allocation at all — the lens we use on destination planning today, and the basis for the planned FBIP tensor type. pdf

  • Giuseppe Castagna, Programming with Union, Intersection, and Negation Types (2022). The semantic-subtyping theory the set-theoretic type system stands on — unions, intersections, negations, and the precise typing of branching and pattern matching. pdf

  • Giuseppe Castagna, Guillaume Duboc & José Valim, The Design Principles of the Elixir Type System (Programming, 2024). The Castagna line — referenced above — applied directly to the Elixir/Erlang family: function domains tied to arity, guard analysis, unified records and dictionaries, and the dynamic type. fz reads like Elixir and types like this. pdf

A note on the name

The name is two keystrokes. The project needed to be called something. If "fz" ends up meaning something later, it'll be because the work earned it, not because the name promised it.

About

No description, website, or topics provided.

Resources

Stars

8 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages