Skip to content
jplevyakPublic

About

Native code compiler for Python wthout type annotations.

Resources

Stars

4 stars

Watchers

4 watching

Forks

Latest commit

 

History

1,994 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pyc — Python Compiler

Website: pyc-lang.org · Releases

pyc is a whole-program, statically typed compiler for Python 3. It infers a type for every value in the program with interprocedural flow analysis (IFA), then emits C (or LLVM IR) and compiles it to a stand-alone native binary with no Python runtime dependency.

The goal is CPython semantics for programs that can be typed without boxing: a compiled program prints what CPython prints, or pyc refuses it at compile time. Where pyc accepts a Python idiom that strict static typing cannot express exactly, the accommodation is named, warns when it changes behaviour, and is refused under --strict (see Modes).

Features

  • Type inference with no annotations. IFA is a simultaneous data- and control-flow analysis by abstract interpretation over a type lattice. It starts from one contour per function and per allocation site, and splits contours only when a type distinction demands it. Each resulting specialization of a function or class is compiled separately.
  • Two backends. The C backend (the default) emits C compiled by the system C compiler at -O2. The LLVM backend (-b) emits LLVM IR directly. It also powers a JIT (-j) and an interactive REPL (--repl). CI runs the whole test suite on both backends.
  • Boehm GC for memory management.
  • Its own parser. Python 3 syntax is parsed by a DParser grammar. CPython isn't needed at compile time or at run time.

Supported Python

  • Classes: inheritance, super(), class attributes, operator overloading (__add__, __getitem__, __eq__, …), __str__/__repr__, __iter__/__next__, __enter__/__exit__, @property with setters and deleters, property(...).
  • Functions: closures, nonlocal/global, lambdas, default and mutable default arguments, *args, **kwargs, keyword-only parameters, f(*t) / f(**d) forwarding, nested functions, local classes.
  • Exceptions: try/except/else/finally, raise, user exception hierarchies, and CPython's built-in exceptions raised by the runtime (ValueError from int("x"), IndexError, KeyError, StopIteration, …).
  • Generators: yield, yield from, send(), and return values.
  • async/await, run on pyc's own event loop (there is no asyncio module).
  • Control flow: if/elif/else, for/while with break/continue/else, with, match/case (all PEP 634 pattern kinds), comprehensions and generator expressions, starred assignment (a, *b = xs).
  • Built-in types: int (64-bit), float, bool, str, bytes, bytearray, list, tuple, dict, set, range, slice, None, with %-formatting and f-strings.
  • Modules: import, from … import, packages and relative imports, all resolved at compile time. Partial implementations of common standard-library modules live in pyc_lib/, including math, random, time, sys, os and os.path, pathlib, fnmatch, textwrap, collections, itertools, functools, bisect, operator, heapq, copy, struct, string, socket, logging and part of re (issues/207 lists what is not there). File I/O uses open(), which takes a str or a pathlib path.
  • Type hints and dataclasses: annotations are accepted and ignored, typing provides its names (TYPE_CHECKING blocks are skipped, TypeAlias assignments dropped), and @dataclass generates __init__, __repr__, __eq__ and __hash__ as CPython does, with field() defaults and factories, frozen=True and __post_init__.
  • C libraries: a static subset of ctypes. Declare a function with CDLL, argtypes and restype and it compiles to a direct call, with Structure and POINTER for C data. The same file runs under CPython. See Calling a C library.

Language extensions

pyc exposes a small set of compiler directives. Its builtin library (__pyc__/) is written with them:

Name Purpose
__pyc_c_call__(ret, fn, ...) Inline a raw C function call (programs use ctypes instead)
__pyc_primitive__(sym, ...) Invoke a compiler primitive
__pyc_char__ The uint8 / C char type
__pyc_operator__ Access C-level operators directly
__pyc_insert_c_header__(h) Include a C header in the generated code
@vector("s") Annotate a class as a fixed-size value-type vector

A field that holds different types in different objects needs no declaration: a class whose __init__ stores an int in one instance and a str in another gets one layout per type.

Install

Each release has a self-contained build for Linux (x86-64, arm64) and macOS (Apple silicon, Intel). Unpack it anywhere and put its bin/ on your PATH:

curl -LO https://github.com/jplevyak/pyc/releases/download/v0.1.2/pyc-0.1.2-linux-x86_64.tar.gz
tar xzf pyc-0.1.2-linux-x86_64.tar.gz
export PATH="$PWD/pyc-0.1.2-linux-x86_64/bin:$PATH"
pyc hello.py && ./hello

(For Linux on arm64, linux-aarch64; for macOS, darwin-arm64 or darwin-x86_64. Each release lists the SHA256SUMS of its tarballs.)

The tarball holds bin/pyc, with LLVM and the Boehm GC linked in, and lib/pyc/, pyc's library, which pyc finds beside itself. It needs:

  • clang++ and make, any recent version (pyc compiles the C it generates with them, and links its LLVM output with clang++). On Ubuntu: apt-get install clang make. On macOS, the Xcode Command Line Tools: xcode-select --install.
  • Linux: glibc 2.39 or newer (Ubuntu 24.04, Debian 13, Fedora 40 and later). macOS: 15 (Sequoia) or newer.

The programs pyc builds link a static Boehm GC from the tarball, so they need nothing installed to run. Older Linux distributions build from source for now.

Requirements

To build from source:

  • clang/clang++ (C++23) and llvm-ar
  • LLVM development libraries (llvm-config must be on PATH); the LLVM backend is always built
  • Boehm GC (libgc, libgccpp)
  • dparser, built with GC support and installed

On Ubuntu / Debian:

apt-get install clang llvm-dev libgc-dev

On Fedora / RHEL:

dnf install clang llvm-devel gc-devel

CI uses clang/LLVM 20 on Ubuntu 24.04. The unversioned packages there are LLVM 18, whose coroutine ABI breaks the generator and async tests.

Building

ifa (the IFA library) is included as a git subtree under ifa/ and is built automatically. Build and install dparser separately first:

# 1. Build and install dparser (with GC support)
git clone https://github.com/jplevyak/dparser.git
cd dparser
sudo make install D_USE_GC=1
cd ..

# 2. Clone and build pyc
git clone https://github.com/jplevyak/pyc.git
cd pyc
make

make builds ifa/libifa_gc.a, the runtime and the pyc compiler binary. pyc runs from the checkout as it is (it finds its library beside itself), or make install PREFIX=/usr/local installs bin/pyc and its library in lib/pyc/ (DESTDIR is honoured).

Build options

Variable Effect
DEBUG=1 Adds -g -DDEBUG (default)
OPTIMIZE=1 Builds pyc itself with -O3 -march=native (default; OPTIMIZE= turns it off)
PROFILE=1 Enable profiling with -pg
PORTABLE=1 No -march=native, so the binary runs on any CPU of its architecture (release builds)

make PORTABLE=1 dist, in a clean tree, builds the release tarball: a statically linked pyc and its library, under dist/. ./dist_smoke.sh dist/pyc-*.tar.gz checks it compiles and runs programs on both backends from a fresh directory. The release workflow (.github/workflows/release.yml) does both on every v* tag.

For an unoptimized build to step through in gdb, use make clean && make OPTIMIZE=.

The Makefiles track neither build flags nor all header dependencies. Run make clean after changing a flag or a header; otherwise stale objects can produce failures that look like real bugs.

Optional: a collector with a larger mark stack

Large programs compile faster against a libgc built with a larger initial mark stack. The packaged library works and nothing requires this; on shedskin_examples/othello3 it takes flow analysis from 126 s to 113 s.

bdwgc starts its mark stack at 4096 entries and doubles it once per collection. The analysis graph of a large program needs over a million, and a mark that runs out re-scans the heap, so every collection until then is several times slower than it should be. The size is a build-time constant with no run-time setting (ifa/186).

git clone --depth 1 --branch v8.2.8 https://github.com/ivmai/bdwgc.git
cd bdwgc
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DBUILD_SHARED_LIBS=ON -Denable_cplusplus=OFF -Dbuild_tests=OFF -Denable_docs=OFF \
      -DCFLAGS_EXTRA="-DINITIAL_MARK_STACK_SIZE=4194304"
cmake --build build -j
sudo cmake --install build && sudo ldconfig

Then rebuild pyc from clean (make clean && make), since the installed headers changed. ldd ./pyc | grep libgc should name /usr/local/lib/libgc.so.1, and GC_PRINT_STATS=1 ./pyc big.py 2>&1 | grep -c "Mark stack overflow" should print 0.

The stack is 64 MiB of address space, touched only as far as a mark reaches, so a small compile is no slower and no larger. -Denable_cplusplus=OFF leaves libgccpp to the distribution's package. Compiled programs link the same libgc and get the same stack.

Usage

pyc [options] <file.py>

pyc reads <file.py> and everything it imports, analyzes the whole program, and writes a native executable named after the input with its extension stripped (hello.py → hello), unless -o names one. The intermediate C is kept as <file.py>.c.

pyc finds its builtin library (__pyc__/, pyc_lib/ and the runtime) by itself: in -D <dir> or $PYC_SYSTEM_DIRECTORY when given, else the current directory if it has __pyc__/, else beside the pyc executable (through any symlink, so a checkout's pyc symlinked onto your PATH works), else in ../lib/pyc from the executable, the installed layout.

Running a script directly

#!/usr/bin/env -S pyc -x
import sys
print("hello", sys.argv[1:])

With that first line and the executable bit set, ./hello.py a b compiles the script once and runs it, passing it a b. -x keeps the executable in $XDG_CACHE_HOME/pyc (default ~/.cache/pyc), in a directory named after the script and a hash of its content, its path, the pyc binary and the options. The directory also holds a manifest of every source file the compile read (the script, its imports, pyc_lib modules and the builtin library) with their content hashes. A later run whose sources are all unchanged starts the cached executable directly, in a few milliseconds; if any changed, it recompiles. Nothing is written beside the script, its exit status is the program's, and sys.argv[0] is the script's path, as under CPython. Other options go before -x's script, as in #!/usr/bin/env -S pyc -b -x.

Options

Flag Description
-D <dir>, --system-directory <dir> pyc's library directory (default $PYC_SYSTEM_DIRECTORY, else ., beside the executable, or ../lib/pyc from it)
-o <path>, --output <path> Output binary path (default: input filename, extension stripped)
--cflags <flags> Extra flags for compiling the generated program (also $PYC_CFLAGS)
--libs <flags> Extra flags for linking it, e.g. --libs "-L/opt/lib -lfoo" (also $PYC_LIBS)
-x, --exec Compile (cached in ~/.cache/pyc) and run; arguments after the script go to the program. For #! lines; see above
--strict CPython semantics or a compile error: every type violation is fatal, no accommodations
--permissive The default: named accommodations are allowed, each with a warning
--safe Permissive, and also zero-initialize locals that may be read before assignment
--refuse-imprecise Refuse a program whose inference left a container element or parameter untyped
-b, --emit-llvm Use the LLVM backend
-j, --jit Compile with the LLVM backend and run in-process (implies -b)
--repl Interactive REPL (uses the JIT)
-O, --optimize Compile the generated C at -O2 (the default)
--no-optimize Compile the generated C at -O0, for debugging
-g, --debug-info Emit debug information
--html Emit an HTML visualization of the type-annotated program
-v, --verbose / -d, --debug Increase verbosity / debug output (repeat for more)
--version, --license, -h, --help Version, license, help (--help also lists development options)

Modes

Static typing without boxing cannot express everything CPython does. pyc's rule is that it never deviates silently:

  • --strict: the program has CPython's semantics, or it does not compile. Gaps in this guarantee are bugs, tracked as silent wrong answers in issues/.
  • --permissive (the default) additionally allows a few named accommodations. Examples: a numeric slot that holds both int and float is widened to float, and a generator's next() resume delivers 0 instead of None when the generator is also sent numbers. Each one is behind a named flag, warns at the place it applies, and is off under --strict.

A value that genuinely holds two unrelated types, such as an int in one branch and a str in the other, has no representation. pyc refuses it in every mode and names the variable and the union.

Module search path

Imports resolve against the current directory, then <pyc-dir>/pyc_lib/, then each directory in PYTHONPATH. The source file's own directory is not searched, so compile from the program's directory, or add it to PYTHONPATH:

PYTHONPATH=/my/libs pyc -D ~/pyc myprogram.py

Examples

Hello world

# hello.py
print("Hello, world!")
pyc -D ~/pyc hello.py && ./hello

Fibonacci

def fib(x):
    if x == 0 or x == 1:
        return 1
    else:
        return fib(x-2) + fib(x-1)

print(fib(33))

Classes and inheritance

class A(object):
    n = 2
    def __init__(self, a):
        print(a + 10)
    def method(self):
        return self.n

class B(A):
    def __init__(self, a):
        super(B, self).__init__(a)
        print(self.n)

y = B(3)

List comprehensions

x = [1, 2, 3]
y = [5, 6, 7]
z = [i + j + 1 for i in x for j in y]
print(z)

Closures and default arguments

def f(a, L=[]):
    L.append(a)
    return L

print(f(1))   # [1]
print(f(2))   # [1, 2]
print(f(3))   # [1, 2, 3]

Polymorphic fields

A field may hold a different type in each object. No declaration is needed: each type gets its own layout of C.

class C:
    def __init__(self, val):
        self.value = val

print(C(1).value)
print(C("hello").value)
print(C(3.14).value)

Calling a C library

Declare a C function the way CPython's ctypes asks. pyc reads the declarations at compile time, emits a direct call and links the library. The file runs unchanged under CPython.

import ctypes
import ctypes.util

libm = ctypes.CDLL(ctypes.util.find_library("m"))
libm.hypot.argtypes = [ctypes.c_double, ctypes.c_double]
libm.hypot.restype = ctypes.c_double

libc = ctypes.CDLL(None)
libc.strlen.argtypes = [ctypes.c_char_p]
libc.strlen.restype = ctypes.c_size_t

print(libm.hypot(3.0, 4.0))    # 5.0
print(libc.strlen(b"hello"))   # 5

An argument of the wrong type is a compile-time error, where CPython raises ArgumentError at run time. Structure with _fields_, POINTER, byref and pointer are supported too. For a library outside the default search path, pass --cflags and --libs. The supported subset, what is refused and what is not done yet are in issues/188.

Testing

make test          # everything CI runs, in CI's order, on both backends (~4 minutes)

make test chains four groups, each of which also runs on its own:

make test-core       # ifa unit tests, IR phase tests, and the end-to-end suite
                     # (./test_pyc.py) on the C backend, then on LLVM
make test-ifa-llvm   # ifa V-language LLVM smoke (= make -C ifa test_llvm)
make test-dparse     # DParser parse-only validation of every test file
make test-links      # every relative Markdown link resolves (needs no build)

The end-to-end suite compiles each tests/*.py, runs it, and compares its output with the expected output and with CPython's. When green it reports 524 passed and 16 known issues on each backend. Tests carrying a .known_issue sidecar are filed bugs with CPython's answer as the expected output; they report KNOWN and flip to PASS once fixed. make test stops at the first failing group, so check its exit status rather than reading the tail of its output.

Beware a name trap: this directory's make test_llvm is the pyc end-to-end suite on the LLVM backend, while make -C ifa test_llvm is the V-language smoke (make test-ifa-llvm).

The shedskin corpus

shedskin_examples/ holds shedskin's 77 example programs, pyc's larger benchmark. ./corpus_sweep.sh -m check compiles and runs each one and compares its output with CPython's. Results are cached under sweeps/, keyed on the source tree. As of 2026-11-10:

  • All 77 programs compile. The largest, othello3, takes about 170 s; it exceeded the 400 s compile cap until call matching and the stages after flow analysis were made cheaper (ifa/207, ifa/186).
  • All 77 binaries run to completion.
  • 59 print output byte-identical to CPython's, and 1 (othello2) differs.
  • The other 17 can't be compared because CPython itself exceeds the 120 s run cap.

Architecture

Python source (.py)
  → DParser (python.g)          — parse to PyDAST
  → build_syms                  — symbol tables, scopes, classes
  → build_if1                   — lower to IF1, ifa's intermediate form
  → IFA flow analysis           — type inference with demand-driven contour splitting
  → clone                       — one concrete function/type per contour
  → optimize                    — dead code, inlining, constant folding
  → C codegen  → cc → binary
    or LLVM codegen → binary / JIT

The IFA engine (ifa/) is a general-purpose interprocedural analysis library that pyc drives through the IFACallbacks interface. Every value is typed by the set of concrete types that can reach it.

Where to read next:

  • PIPELINE.md: phase-by-phase map from pyc file.py to the binary
  • PYTHON_FRONTEND.md: the Python lowering
  • RUNTIME.md: the C runtime and the __pyc__/ builtin library
  • AGENTS.md: the design rules (demand-driven splitting, CPython semantics, no boxing) and the change-acceptance process
  • ifa/INDEX.md: the IFA library's own documentation

Limitations

pyc compiles a large subset of Python 3, not all of it. Bugs and gaps are tracked in issues/ (frontend, runtime, harness) and ifa/issues/ (analysis and codegen). Known unsupported features include:

  • Dynamic features: eval, exec, and runtime reflection are not supported.
  • Values that need boxing. A variable, field or container element that genuinely holds two unrelated types over time ({int, str}, {None, int}) has no representation and is refused.
  • Argument unpacking: f(*t) binds a tuple's elements to any parameters, and f(**d) works when the callee takes **kwargs. Refused: * of a list or of tuples of different lengths into named parameters, **d into named parameters (so the forwarding decorator fn(*a, **kw)), and a **kwargs whose values have different types in one call (issues/189).
  • Decorators: a @property under another decorator, @x.getter, and a decorated method that takes *args or uses super() are refused (issues/007).
  • complex is not supported.
  • __name__ on a class (C.__name__, cls.__name__) is refused (issues/190).
  • Generators: StopIteration.value read in an except clause is refused when the program raises StopIteration with more than one value type (issues/172).
  • The standard library is limited to the partial modules in pyc_lib/.
  • typing and dataclasses: NamedTuple, TypedDict, Generic and Protocol are not provided. @dataclass must be on a class at module level, a subclass of a dataclass must be one itself, and order, kw_only, slots, InitVar, fields(), asdict() and replace() are refused (issues/195).
  • logging covers loggers, levels, basicConfig, %-style Formatters, Filters and the stream, file and null handlers. A format naming the caller's location (%(lineno)d, %(funcName)s, …), exception(), exc_info/extra, the {/$ styles, LoggerAdapter and logging.handlers are refused (issues/197).
  • os, os.path and pathlib are POSIX only, and take str paths or path objects, not bytes. An OSError carries its message but not errno, strerror or filename. pathlib has no Windows flavour and no as_uri, owner, group or chmod, and Path.open() is text-only; read_bytes(), write_bytes() and open(path, 'rb') work (issues/202).
  • argparse is refused at the import: the attributes of its parse_args() result are named and typed at run time. getopt and sys.argv work (issues/206).
  • Non-ASCII text: a str is stored as UTF-8 bytes, and len, indexing, slicing and iteration count bytes, not characters (issues/203).
  • ctypes is a static subset: structures, pointers, arrays, scalar out-parameters and CFUNCTYPE callbacks (through libffi), but no unions, bit fields or cast; declarations must be at module top level, and a program with foreign calls does not run under --jit (issues/188).
  • Class attributes stored through the class after it is used: an instance or a subclass keeps the value the class had when the instance or subclass was made, so C.k += 1 is not seen through self.k, nor A.n = 4 through B.n. CPython looks the attribute up when it is read. A silent difference (issues/173 item 1).
  • Cross-class method assignment: a.x = b.x, where a and b are instances of unrelated classes, is refused at compile time (issues/180).
  • A known silent wrong answer under repair: an out-of-range sequence index does not raise IndexError (issues/179).

License

BSD 3-Clause — see LICENSE.

Copyright (c) John Bradley Plevyak.

About

Native code compiler for Python wthout type annotations.

Resources

Stars

4 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

Languages