litgo: literate Go

tangles to 51 files

You are reading a program. Not the documentation for a program, not a tour of one, the program itself. Every source file in this repository (the Go module, its tests, the Neovim plugin) is tangled out of the Markdown in front of you. The only file written by hand is the README.

Donald Knuth called this literate programming back in 1984. The code is written for a human first. You put it in whatever order explains it best, in pieces small enough to talk about one at a time, and a tool puts the pieces back in the order the compiler wants. litgo does four things with a document like that:

flowchart LR
    doc[litgo.lit.md] -->|tangle| src[Go and Lua source]
    doc -->|weave| page[PDF or web page]
    doc -->|prove| qed[contracts and invariants, checked]
    src -->|go build| bin[bin/litgo]
    bin -->|run| doc

Look at the bottom of that picture. The last arrow points from the binary back into the document. That loop is how litgo gets built. The generated sources are committed, so a fresh clone builds with nothing but Go, and from then on litgo rebuilds itself:

go build -o bin/litgo .        # stage 1, from the committed sources
bin/litgo run litgo.lit.md     # tangle, prove, vet, test, rebuild bin/litgo

litgo also takes its own medicine. Every editor feature in this document rests on one small piece of arithmetic, the one that moves a column between the Markdown and the tangled file. If that arithmetic is wrong, everything is wrong, quietly. So it is proved, by the verifier in this document, every time the line above runs. Two halves of one map says what is proved and, just as carefully, what is only tested.

litgo run does whatever the run: directive on the first line says. (It names the packages instead of saying ./..., because running two of the examples leaves two main functions in examples/, which is fine for them and not for go vet.) Now the fun part. Tangle this document with the binary it produces, and you get back exactly the sources that binary was built from. The snake eats its own tail and comes out the same length. That fixed point is how we know the bootstrap works.

go.mod

litgo depends on nothing but the standard library.

module github.com/tlehman/litgo

go 1.24
.gitignore
/bin/
*.pdf
*.html
*.woven.md
/examples/*.go

The format

A .lit.md file is plain Markdown. Everything litgo needs hides in two places: HTML comments, which Markdown renderers don’t show, and code comments, which keep the code valid for highlighters and formatters. Here’s a complete program:

<!-- package: main -->
<!-- imports: fmt -->

# Hello

```go
func main() {
    // <<greet>>
}
```

The greeting gets a section of its own.

<!-- chunk: greet -->
```go
fmt.Println("hello,", /*<<whom>>*/)
```

<!-- chunk: whom -->
```go
"world"
```

Unnamed blocks in the file’s language get written out in the order they appear. Named blocks only show up where something references them. You name a block with a chunk: directive above it, and you reference it with a comment that holds nothing but the name in double angle brackets. A line comment on a line of its own splices the chunk in at that indentation. A block comment splices it into the middle of a line.

Splicing is plain text substitution, on purpose. A chunk is not a function. It sees every name that is in scope wherever it lands. Idiomatic Go is full of big statements (a for around a select around an if, with a defer on top), and you can’t carve those into functions without passing half the local variables around. A chunk lets you lift one part out, explain it somewhere else, and leave the program exactly as it was.

Here are the directives:

Directive Meaning
file: path Starts writing another file. The unnamed blocks in the language of its extension go into it.
package: name Gives the current file a package clause.
imports: a, b, alias c Gives the current file an import block.
chunk: name Names the next block. Blocks with the same name get concatenated.
notangle The next block is only an illustration and stays out of the program.
verbatim In the next block, text that looks like a reference is just text.
tangler-exclude: regexp Drops the source lines that match.
tangler: lang, output: path Set the language and path of the default file. The defaults are go and name.go for name.lit.md.
run: command What litgo run runs after tangling. Without it, litgo run builds and runs the default file.

litgo never reformats the code it tangles. Every output line comes from exactly one source line, so any position in the output can be traced back home.

Specifications need no directive at all, because they’re part of the code. A comment that starts with //@ is a VeGo annotation, and it travels wherever the Go it describes travels, through any number of chunks. A function’s contract can sit in the block that introduces the function while its loop invariant sits three sections later, right next to the paragraph that explains the loop. The verifier sees them joined up, the same way the compiler sees the code. Proofs is the chapter about them, and A tutorial in proofs is the guided tour.

The document model

internal/lit/doc.go

Package lit is the format itself. It parses documents, tangles them while keeping a source map, and works out the structural edits an editor asks for. Every line and column in this package is 0-based, and columns count bytes. That’s what the Go compiler and Neovim use too, once you subtract one from theirs.

A problem can come from the document’s structure, the Go parser, the compiler, or the running program. It doesn’t matter where. Every problem ends up as a Diag pointing into the .lit.md file. The severity values are the ones vim.diagnostic uses.

// Severity of a diagnostic, matching vim.diagnostic.severity.
const (
    SevError   = 1
    SevWarning = 2
    SevInfo    = 3
)

// Diag is a problem located in a .lit.md source file.
type Diag struct {
    File     string `json:"file,omitempty"`
    Line     int    `json:"line"`
    Col      int    `json:"col"`
    EndCol   int    `json:"end_col,omitempty"`
    Severity int    `json:"severity"`
    Message  string `json:"message"`
    Source   string `json:"source"`
}

A directive is an HTML comment on a line by itself. litgo remembers the column where its value starts, because errors can point into directives too. An unused import, for example, gets reported right on its entry in the imports: line.

// Directive is a `<!-- key: value -->` line outside of any code fence.
type Directive struct {
    Key      string
    Value    string
    Line     int
    ValueCol int // byte column where Value starts
}

// Import is one entry of an `imports:` directive.
type Import struct {
    Alias string
    Path  string
    Line  int
    Col   int // column of the entry within the directive line
    Len   int
}

A document writes one or more files. There’s always a default file named after the document, and every file: directive starts another. package: and imports: apply to whichever file is current.

// File is one tangled output. A document always has a default file, named
// after the document; each `file:` directive starts another.
type File struct {
    Path    string // as written; relative paths are relative to the document
    Lang    string
    Line    int // line of the file directive, -1 for the default file
    Package string
    PkgDir  *Directive
    Imports []Import
}

A block remembers which file was current when it showed up, its chunk name if it has one, and which lines its fences sit on. If a fence is indented inside a list, that indentation gets stripped from the block’s content.

// Block is a fenced code block.
type Block struct {
    Lang      string
    File      *File  // the file being written when the block appears
    Chunk     string // chunk name; "" means a root block
    ChunkLine int    // line of the chunk directive, -1 if none
    NoTangle  bool
    Verbatim  bool // text that looks like a reference is just text
    Open      int  // line of the opening fence
    Close     int  // line of the closing fence (len(Lines) if unterminated)
    Strip     int  // fence indentation, stripped from each content line
}

// First and Last content lines (Last is inclusive; Last < First if empty).
func (b *Block) First() int { return b.Open + 1 }
func (b *Block) Last() int  { return b.Close - 1 }

// Doc is a parsed .lit.md document.
type Doc struct {
    Path       string
    Lines      []string
    Directives []Directive
    Blocks     []*Block
    Files      []*File // Files[0] is the default file

    Run      string
    Excludes []*regexp.Regexp
    Diags    []Diag
}

Recognising the syntax

The Markdown side needs two regular expressions. The directive pattern is anchored at both ends and matches its value lazily, so the value can contain --> itself. (An exclude pattern for HTML comments needs exactly that.)

The reference patterns speak the comment syntax of three language families: C-like (//, /* */), Lua and SQL (--, --[[ ]]) and the shell family (#). They capture the comment opener, so renaming a chunk can put the same opener back.

var (
    directiveRe = regexp.MustCompile(`^\s*<!--\s*([A-Za-z][A-Za-z0-9-]*)\s*(?::\s*(.*?))?\s*-->\s*$`)
    fenceRe     = regexp.MustCompile("^( {0,3}|\t*)(```+|~~~+)\\s*([^\\s`]*)(.*)$")

    // RefLineRe matches a whole-line chunk reference: a line comment holding
    // nothing but <<name>>, in the comment syntax of C-like languages, of Lua
    // and SQL, or of the shell family.
    RefLineRe = regexp.MustCompile(`^(\s*)(//|--|#)\s*<<\s*(.+?)\s*>>\s*$`)
    // RefInlineRe matches an inline chunk reference: a block comment.
    RefInlineRe = regexp.MustCompile(`(/\*|--\[\[)\s*<<\s*(.+?)\s*>>\s*(\*/|\]\])`)
)

var knownDirectives = map[string]bool{
    "tangler": true, "package": true, "imports": true, "tangler-exclude": true,
    "output": true, "file": true, "run": true, "chunk": true, "notangle": true, "verbatim": true,
}

A file’s extension picks its language, and the fences of its blocks have to name that language. Most extensions already are the language name. This table covers the ones that aren’t.

// languages maps a file extension to the fence language of the blocks that
// belong in such a file. Anything else uses the extension itself.
var languages = map[string]string{
    ".mod": "gomod", ".md": "markdown", ".py": "python", ".rb": "ruby", ".rs": "rust",
    ".js": "javascript", ".ts": "typescript", ".yml": "yaml", ".bash": "sh",
}

func langOf(path string) string {
    ext := strings.ToLower(filepath.Ext(path))
    if l, ok := languages[ext]; ok {
        return l
    }
    if ext == "" {
        return strings.ToLower(filepath.Base(path)) // Makefile → makefile
    }
    return ext[1:]
}

litgo writes comments as well as reading them. It stamps a marker at the top of each generated file, and :Untangle leaves a reference where the code used to be.

// Comments describes how a language writes the comments that carry chunk
// references and the generated-code marker.
type Comments struct{ Line, Open, Close string }

// CommentsFor returns the comment syntax of a fence language. Languages litgo
// does not know get no marker and no reference syntax of their own.
func CommentsFor(lang string) Comments {
    switch lang {
    case "go", "gomod", "c", "cpp", "java", "javascript", "typescript", "rust", "swift", "kotlin", "zig", "css", "typ", "typst":
        return Comments{"//", "/*", "*/"}
    case "lua":
        return Comments{"--", "--[[", "]]"}
    case "sql", "haskell":
        return Comments{Line: "--"}
    case "sh", "bash", "zsh", "fish", "python", "ruby", "yaml", "toml", "makefile", "make", "dockerfile", "nix", "gitignore":
        return Comments{Line: "#"}
    }
    return Comments{}
}

// NormName canonicalises a chunk name: trimmed, inner whitespace collapsed.
func NormName(s string) string { return strings.Join(strings.Fields(s), " ") }

Parsing

Parsing is one pass over the lines, and Parse never fails. Who cares? The editor does. A document you are editing is broken somewhere almost all the time, and the editor still wants answers about the rest of it.

// Parse reads a .lit.md document. Errors land in Diags.
func Parse(path string, src []byte) *Doc {
    text := strings.ReplaceAll(string(src), "\r\n", "\n")
    text = strings.TrimSuffix(text, "\n")
    d := &Doc{Path: path}
    if text != "" || len(src) > 0 {
        d.Lines = strings.Split(text, "\n")
    }
    cur := &File{Lang: "go", Line: -1}
    d.Files = []*File{cur}

    var pending *Directive // chunk directive awaiting its block
    noTangle, verbatim := false, false
    for i := 0; i < len(d.Lines); i++ {
        line := d.Lines[i]
        if m := fenceRe.FindStringSubmatch(line); m != nil {
            // <<record a code block and skip past it>>
            continue
        }
        if m := directiveRe.FindStringSubmatchIndex(line); m != nil {
            key := strings.ToLower(line[m[2]:m[3]])
            if knownDirectives[key] {
                // <<record a directive>>
                continue
            }
        }
        // <<prose cancels directives that were waiting for a block>>
    }
    if pending != nil {
        d.diag(pending.Line, 0, SevWarning,
            fmt.Sprintf("chunk directive %q is not followed by a code block", NormName(pending.Value)))
    }
    return d
}

A fence closes at the next line made of the same fence character, at least as long. That’s how a four-backtick block can hold three-backtick fences, and how this document can hold its own example. Any chunk:, notangle or verbatim directives that were waiting get attached to this block.

⟨record a code block and skip past it⟩ ≡ used in the program
b := &Block{Lang: strings.ToLower(m[3]), File: cur, Open: i, ChunkLine: -1, Strip: len(m[1])}
b.Lang = strings.Trim(b.Lang, "{}.")
marker := m[2]
b.Close = len(d.Lines)
for j := i + 1; j < len(d.Lines); j++ {
    t := strings.TrimSpace(d.Lines[j])
    if strings.HasPrefix(t, marker) && strings.Trim(t, marker[:1]) == "" {
        b.Close = j
        break
    }
}
if b.Close == len(d.Lines) {
    d.diag(i, 0, SevWarning, "code fence is never closed")
}
if pending != nil {
    b.Chunk, b.ChunkLine = NormName(pending.Value), pending.Line
}
b.NoTangle, b.Verbatim = noTangle, verbatim
pending, noTangle, verbatim = nil, false, false
d.Blocks = append(d.Blocks, b)
i = b.Close

Directives that belong to the next block wait until that block shows up. file: switches the current file. The rest configure the document or the current file.

⟨record a directive⟩ ≡ used in the program
dir := Directive{Key: key, Line: i, ValueCol: m[3]}
if m[4] >= 0 {
    dir.Value, dir.ValueCol = line[m[4]:m[5]], m[4]
}
d.Directives = append(d.Directives, dir)
switch key {
case "chunk":
    if NormName(dir.Value) == "" {
        d.diag(i, 0, SevError, "chunk directive needs a name")
    } else {
        p := dir
        pending = &p
    }
case "notangle":
    noTangle = true
case "verbatim":
    verbatim = true
case "file":
    if p := strings.TrimSpace(dir.Value); p == "" {
        d.diag(i, 0, SevError, "file directive needs a path")
    } else {
        cur = &File{Path: p, Lang: langOf(p), Line: i}
        d.Files = append(d.Files, cur)
    }
default:
    d.apply(&d.Directives[len(d.Directives)-1], cur)
}

Blank lines are allowed between a directive and its block. Prose is not. A chunk: directive followed by a paragraph names nothing, and litgo warns you instead of quietly naming some block further down.

⟨prose cancels directives that were waiting for a block⟩ ≡ used in the program
if strings.TrimSpace(line) != "" {
    if pending != nil {
        d.diag(pending.Line, 0, SevWarning, fmt.Sprintf("chunk directive %q is not followed by a code block", NormName(pending.Value)))
    }
    pending, noTangle, verbatim = nil, false, false
}
func (d *Doc) diag(line, col, sev int, msg string) {
    d.Diags = append(d.Diags, Diag{Line: line, Col: col, Severity: sev, Message: msg, Source: "litgo"})
}

// apply handles the directives that configure the document or the file
// currently being written.
func (d *Doc) apply(dir *Directive, cur *File) {
    switch dir.Key {
    case "tangler":
        d.Files[0].Lang = strings.ToLower(strings.TrimSpace(dir.Value))
    case "output":
        d.Files[0].Path = strings.TrimSpace(dir.Value)
    case "package":
        cur.Package = strings.TrimSpace(dir.Value)
        cur.PkgDir = dir
    case "run":
        d.Run = strings.TrimSpace(dir.Value)
    case "tangler-exclude":
        re, err := regexp.Compile(dir.Value)
        if err != nil {
            d.diag(dir.Line, dir.ValueCol, SevError, "bad tangler-exclude pattern: "+err.Error())
            return
        }
        d.Excludes = append(d.Excludes, re)
    case "imports":
        // Entries are comma separated; each is `path` or `alias path`.
        off := 0
        for _, part := range strings.Split(dir.Value, ",") {
            entry := strings.TrimSpace(part)
            col := dir.ValueCol + off + strings.Index(part, entry)
            off += len(part) + 1
            if entry == "" {
                continue
            }
            imp := Import{Line: dir.Line, Col: col, Len: len(entry)}
            f := strings.Fields(entry)
            imp.Path = strings.Trim(f[len(f)-1], `"`)
            if len(f) > 1 {
                imp.Alias = f[0]
            }
            cur.Imports = append(cur.Imports, imp)
        }
    }
}

Files, blocks and chunks

OutPath is where a file gets written. The default file is named after the document: name.lit.md tangles to name.go.

func (d *Doc) OutPath(f *File) string {
    dir := filepath.Dir(d.Path)
    if f.Path != "" {
        if filepath.IsAbs(f.Path) {
            return f.Path
        }
        return filepath.Join(dir, f.Path)
    }
    base := filepath.Base(d.Path)
    base = strings.TrimSuffix(base, ".md")
    base = strings.TrimSuffix(base, ".lit")
    ext := f.Lang
    if ext == "" {
        ext = "txt"
    }
    return filepath.Join(dir, base+"."+ext)
}

A block can take part in tangling in two ways. Either it’s written straight into its file, or it’s a chunk. A chunk can be in any language, because where it lands depends on who references it.

// IsRoot reports whether the block is written straight into its file: it is
// unnamed and in the file's language.
func (b *Block) IsRoot() bool { return b.Chunk == "" && !b.NoTangle && b.Lang == b.File.Lang }

// Tangled reports whether the block takes part in tangling, as a root block
// or as a chunk.
func (b *Block) Tangled() bool { return !b.NoTangle && (b.Chunk != "" || b.IsRoot()) }

// BlockAt returns the block whose fences or content include the line.
func (d *Doc) BlockAt(line int) *Block {
    for _, b := range d.Blocks {
        if line >= b.Open && line <= b.Close {
            return b
        }
    }
    return nil
}
// Ref is one use of a chunk.
type Ref struct {
    Name   string
    Line   int
    Col    int // start of the reference syntax
    EndCol int
    Inline bool
    Indent string
    Open   string // comment opener as written, reused when the reference is renamed
    Close  string
}

// Text spells a reference to name in the same comment syntax.
func (r Ref) Text(name string) string {
    if r.Inline {
        return r.Open + "<<" + name + ">>" + r.Close
    }
    return r.Open + " <<" + name + ">>"
}

// RefsIn lists the chunk references on one source line.
func RefsIn(line string, lineNo int) []Ref {
    if m := RefLineRe.FindStringSubmatchIndex(line); m != nil {
        return []Ref{{Name: NormName(line[m[6]:m[7]]), Line: lineNo, Col: m[3], EndCol: len(strings.TrimRight(line, " \t")),
            Indent: line[m[2]:m[3]], Open: line[m[4]:m[5]]}}
    }
    var out []Ref
    for _, m := range RefInlineRe.FindAllStringSubmatchIndex(line, -1) {
        out = append(out, Ref{Name: NormName(line[m[4]:m[5]]), Line: lineNo, Col: m[0], EndCol: m[1], Inline: true,
            Open: line[m[2]:m[3]], Close: line[m[6]:m[7]]})
    }
    return out
}

// refs lists the references a block makes; a verbatim block makes none.
func (d *Doc) refs(b *Block, line int) []Ref {
    if b.Verbatim {
        return nil
    }
    return RefsIn(d.Lines[line], line)
}

Chunks is the index everything else leans on: tangling, the checks for unused and undefined chunks, renaming, and navigation in the editor.

// Chunk gathers every block sharing a name, plus all references to it.
type Chunk struct {
    Name   string
    Blocks []*Block
    Refs   []Ref
}

// Chunks indexes named chunks in document order. References to undefined
// chunks appear as entries with no Blocks.
func (d *Doc) Chunks() []*Chunk {
    byName := map[string]*Chunk{}
    var order []*Chunk
    get := func(name string) *Chunk {
        c := byName[name]
        if c == nil {
            c = &Chunk{Name: name}
            byName[name] = c
            order = append(order, c)
        }
        return c
    }
    for _, b := range d.Blocks {
        if b.Tangled() && b.Chunk != "" {
            c := get(b.Chunk)
            c.Blocks = append(c.Blocks, b)
        }
    }
    for _, b := range d.Blocks {
        if !b.Tangled() {
            continue
        }
        for i := b.First(); i <= b.Last(); i++ {
            for _, r := range d.refs(b, i) {
                c := get(r.Name)
                c.Refs = append(c.Refs, r)
            }
        }
    }
    return order
}

Tangling

internal/lit/tangle.go

Tangling produces output lines, and for every line it records where the bytes came from. That record is the whole trick. A line spliced together from a host line and an inline chunk has several segments, and an error column gets looked up in them.

// Seg says that Len bytes of an output line starting at OutCol came from
// (SrcLine, SrcCol) of the literate source. Via lists the reference lines the
// text was interpolated through, outermost first.
type Seg struct {
    OutCol, Len     int
    SrcLine, SrcCol int
    Via             []int
}

// OutLine is one line of tangled output.
type OutLine struct {
    Text string
    Segs []Seg
}

// Output is one tangled file. Its first Header lines are the ones litgo
// generates: the marker, the package clause, the imports.
type Output struct {
    File   *File
    Path   string
    Lines  []OutLine
    Header int
}

// Result of tangling one document.
type Result struct {
    Doc   *Doc
    Files []*Output
    Diags []Diag
}

Every file litgo writes starts with a marker in that language’s comment syntax, and litgo refuses to overwrite a file without one. Why so cautious? Because the very first thing a new user does is run litgo tangle pool.lit.md right next to a hand-written pool.go.

// generatedMarker is the text of the first line of every file litgo writes,
// after the comment leader; tangle refuses to overwrite files that lack it.
const generatedMarker = " Code generated by litgo from "

// Generated reports whether file content starts with litgo's marker.
func Generated(content []byte) bool {
    first, _, _ := strings.Cut(string(content), "\n")
    for _, leader := range []string{"//", "--", "#"} {
        if strings.HasPrefix(first, leader+generatedMarker) {
            return true
        }
    }
    return false
}
// Tangle expands the document into its output files.
func (d *Doc) Tangle() *Result {
    r := &Result{Doc: d}
    r.Diags = append(r.Diags, d.Diags...)
    t := &tangler{d: d, r: r, chunks: map[string]*Chunk{}, reported: map[string]bool{}}
    for _, c := range d.Chunks() {
        t.chunks[c.Name] = c
    }

    for _, f := range d.Files {
        var body []OutLine
        roots := 0
        for _, b := range d.Blocks {
            if b.File != f || !b.IsRoot() {
                continue
            }
            roots++
            lines := t.block(b, nil, nil)
            if len(lines) == 0 {
                continue
            }
            if len(body) > 0 {
                body = append(body, OutLine{})
            }
            body = append(body, lines...)
        }
        if roots == 0 {
            if f.Line >= 0 {
                r.Diags = append(r.Diags, Diag{Line: f.Line, Severity: SevWarning, Source: "litgo",
                    Message: fmt.Sprintf("no unnamed ```%s blocks follow: %s would be empty", f.Lang, f.Path)})
            }
            continue
        }
        out := &Output{File: f, Path: d.OutPath(f), Lines: d.header(f)}
        out.Header = len(out.Lines)
        if len(out.Lines) > 0 && len(body) > 0 {
            out.Lines = append(out.Lines, OutLine{})
        }
        out.Lines = append(out.Lines, body...)
        r.Files = append(r.Files, out)
    }
    if len(r.Files) == 0 {
        r.Diags = append(r.Diags, Diag{Line: 0, Severity: SevWarning, Source: "litgo",
            Message: fmt.Sprintf("no unnamed ```%s blocks: nothing to tangle (chunks are only emitted where referenced)", d.Files[0].Lang)})
    }
    for _, c := range d.Chunks() {
        if len(c.Blocks) > 0 && len(c.Refs) == 0 {
            r.Diags = append(r.Diags, Diag{Line: c.Blocks[0].ChunkLine, Severity: SevWarning, Source: "litgo",
                Message: fmt.Sprintf("chunk ⟨%s⟩ is never used", c.Name)})
        }
    }
    for i := range r.Diags {
        if r.Diags[i].File == "" {
            r.Diags[i].File = d.Path
        }
    }
    return r
}

// header is what litgo writes before the blocks: the generated-code marker
// and, for a file with a package directive, the package clause and imports.
// Header lines map to the directives they came from, so an unused import is
// reported on its entry in the imports directive.
func (d *Doc) header(f *File) []OutLine {
    var h []OutLine
    if leader := CommentsFor(f.Lang).Line; leader != "" {
        h = append(h, OutLine{Text: leader + generatedMarker + filepath.Base(d.Path) + "; DO NOT EDIT."})
    }
    if f.Package == "" {
        return h
    }
    h = append(h, OutLine{}, OutLine{Text: "package " + f.Package,
        Segs: []Seg{{OutCol: 8, Len: len(f.Package), SrcLine: f.PkgDir.Line, SrcCol: f.PkgDir.ValueCol}}})
    if len(f.Imports) > 0 {
        h = append(h, OutLine{}, OutLine{Text: "import ("})
        // <<put the imports in the order gofmt wants>>
        for i, imp := range imports {
            if i > 0 && external(imp) && !external(imports[i-1]) {
                h = append(h, OutLine{})
            }
            text := "\t"
            if imp.Alias != "" {
                text += imp.Alias + " "
            }
            text += `"` + imp.Path + `"`
            // The path between the quotes is the path in the directive.
            h = append(h, OutLine{Text: text, Segs: []Seg{{OutCol: len(text) - 1 - len(imp.Path), Len: len(imp.Path),
                SrcLine: imp.Line, SrcCol: imp.Col + strings.LastIndex(d.Lines[imp.Line][imp.Col:imp.Col+imp.Len], imp.Path)}}})
        }
        h = append(h, OutLine{Text: ")"})
    }
    return h
}

The generated files are committed, so they should already be clean under gofmt. gofmt wants the standard library imports first, then a blank line, then everything else, each group sorted. A standard library path is easy to spot: its first element has no dot in it.

⟨put the imports in the order gofmt wants⟩ ≡ used in the program
external := func(imp Import) bool { return strings.Contains(strings.Split(imp.Path, "/")[0], ".") }
imports := append([]Import{}, f.Imports...)
sort.SliceStable(imports, func(i, j int) bool {
    if a, b := external(imports[i]), external(imports[j]); a != b {
        return b
    }
    return imports[i].Path < imports[j].Path
})

Expansion is recursive. It carries the stack of chunk names along to catch a chunk that includes itself (a snake that really would eat its own tail, forever), and the stack of reference lines so each segment can record them as its Via.

type tangler struct {
    d        *Doc
    r        *Result
    chunks   map[string]*Chunk
    reported map[string]bool
}

func (t *tangler) errorAt(ref Ref, msg string) {
    key := fmt.Sprintf("%d:%d:%s", ref.Line, ref.Col, msg)
    if t.reported[key] {
        return
    }
    t.reported[key] = true
    t.r.Diags = append(t.r.Diags, Diag{Line: ref.Line, Col: ref.Col, EndCol: ref.EndCol, Severity: SevError, Message: msg, Source: "litgo"})
}

func (t *tangler) excluded(line string) bool {
    for _, re := range t.d.Excludes {
        if re.MatchString(line) {
            return true
        }
    }
    return false
}

// expand returns the lines of a named chunk.
func (t *tangler) expand(ref Ref, stack []string, via []int) []OutLine {
    for _, s := range stack {
        if s == ref.Name {
            t.errorAt(ref, fmt.Sprintf("chunk ⟨%s⟩ includes itself: %s → %s", ref.Name, strings.Join(stack, " → "), ref.Name))
            return nil
        }
    }
    c := t.chunks[ref.Name]
    if c == nil || len(c.Blocks) == 0 {
        t.errorAt(ref, fmt.Sprintf("chunk ⟨%s⟩ is not defined", ref.Name))
        return nil
    }
    stack = append(append([]string{}, stack...), ref.Name)
    via = append(append([]int{}, via...), ref.Line)
    var out []OutLine
    for _, b := range c.Blocks {
        out = append(out, t.block(b, stack, via)...)
    }
    return out
}

References are recognised before excludes are applied, so an exclude pattern for comments can’t swallow a reference.

func (t *tangler) block(b *Block, stack []string, via []int) []OutLine {
    var out []OutLine
    for i := b.First(); i <= b.Last(); i++ {
        raw := t.d.Lines[i]
        strip := 0
        for strip < b.Strip && strip < len(raw) && (raw[strip] == ' ' || raw[strip] == '\t') {
            strip++
        }
        line := raw[strip:]
        refs := t.d.refs(b, i)
        if len(refs) == 1 && !refs[0].Inline {
            indent := refs[0].Indent[min(strip, len(refs[0].Indent)):]
            for _, sub := range t.expand(refs[0], stack, via) {
                out = append(out, indentLine(sub, indent))
            }
            continue
        }
        if t.excluded(line) {
            continue
        }
        if len(refs) == 0 {
            ol := OutLine{Text: line}
            if line != "" {
                ol.Segs = []Seg{{OutCol: 0, Len: len(line), SrcLine: i, SrcCol: strip, Via: via}}
            }
            out = append(out, ol)
            continue
        }
        out = append(out, t.inline(raw, strip, i, refs, stack, via)...)
    }
    return out
}

// inline splices chunks into the middle of a host line. Continuation lines of
// a multi-line chunk inherit the host line's indentation.
func (t *tangler) inline(raw string, strip, lineNo int, refs []Ref, stack []string, via []int) []OutLine {
    hostIndent := raw[strip : len(raw)-len(strings.TrimLeft(raw[strip:], " \t"))]
    cur := OutLine{}
    var out []OutLine
    lit := func(from, to int) {
        if to > from {
            cur.Segs = append(cur.Segs, Seg{OutCol: len(cur.Text), Len: to - from, SrcLine: lineNo, SrcCol: from, Via: via})
            cur.Text += raw[from:to]
        }
    }
    pos := strip
    for _, ref := range refs {
        lit(pos, ref.Col)
        pos = ref.EndCol
        for k, sub := range t.expand(ref, stack, via) {
            if k > 0 {
                out = append(out, cur)
                cur = OutLine{}
                if sub.Text != "" {
                    cur.Text = hostIndent
                }
            }
            shift := len(cur.Text)
            for _, s := range sub.Segs {
                s.OutCol += shift
                cur.Segs = append(cur.Segs, s)
            }
            cur.Text += sub.Text
        }
    }
    lit(pos, len(raw))
    return append(out, cur)
}

func indentLine(l OutLine, indent string) OutLine {
    if l.Text == "" || indent == "" {
        return l
    }
    segs := make([]Seg, len(l.Segs))
    for i, s := range l.Segs {
        s.OutCol += len(indent)
        segs[i] = s
    }
    return OutLine{Text: indent + l.Text, Segs: segs}
}
// Bytes renders the tangled file.
func (o *Output) Bytes() []byte {
    var sb strings.Builder
    for _, l := range o.Lines {
        sb.WriteString(l.Text)
        sb.WriteByte('\n')
    }
    return []byte(sb.String())
}

// HasErrors reports whether tangling itself failed.
func (r *Result) HasErrors() bool {
    for _, d := range r.Diags {
        if d.Severity == SevError {
            return true
        }
    }
    return false
}

From output positions back to the source

The compiler says something like pool.go:31:14, and Map answers with the line and column in the Markdown. A column inside indentation that tangling added doesn’t belong to any segment, so it goes to the nearest one. A generated line (the header, or a blank line between blocks) goes to the nearest mapped line above it.

// Map translates a 0-based output position to the literate source. If the
// exact column is not covered, the nearest segment on the line is used; if
// the line is generated, the nearest mapped line above it.
func (o *Output) Map(line, col int) (srcLine, srcCol int, via []int, ok bool) {
    if line < 0 || line >= len(o.Lines) {
        return 0, 0, nil, false
    }
    for l := line; l >= 0; l-- {
        segs := o.Lines[l].Segs
        if len(segs) == 0 {
            continue
        }
        if l != line {
            s := segs[0]
            return s.SrcLine, s.SrcCol, s.Via, true
        }
        best := segs[0]
        for _, s := range segs {
            if col >= s.OutCol && col < s.OutCol+s.Len {
                return s.SrcLine, s.src(col), s.Via, true
            }
            if s.OutCol <= col {
                best = s
            }
        }
        if col >= best.OutCol+best.Len {
            return best.SrcLine, best.SrcCol + max(best.Len-1, 0), best.Via, true
        }
        return best.SrcLine, best.SrcCol, best.Via, true
    }
    return 0, 0, nil, false
}

Go’s own parser gets through a tangled file in microseconds. So litgo reports syntax errors while you type, long before the compiler is involved.

// SyntaxCheck parses the tangled Go files and reports syntax errors at their
// literate positions. It is fast enough to run on every edit.
func (r *Result) SyntaxCheck() []Diag {
    if r.HasErrors() {
        return nil
    }
    var out []Diag
    seen := map[int]bool{} // the parser cascades; the first error on a line is the real one
    for _, o := range r.Files {
        if o.File.Lang != "go" {
            continue
        }
        fset := token.NewFileSet()
        _, err := parser.ParseFile(fset, o.Path, o.Bytes(), parser.AllErrors|parser.SkipObjectResolution)
        list, _ := err.(scanner.ErrorList)
        for _, e := range list {
            line, col, _, ok := o.Map(e.Pos.Line-1, e.Pos.Column-1)
            if !ok || seen[line] {
                continue
            }
            seen[line] = true
            out = append(out, Diag{File: r.Doc.Path, Line: line, Col: col, Severity: SevError, Message: e.Msg, Source: "syntax"})
            if len(out) >= 20 {
                return out
            }
        }
    }
    return out
}

And from the source to the output

A tool that answers questions about Go (what is this, where is it defined, what could come next) has to be asked about the tangled file, at the spot where the text under your cursor ended up. Locate is Map run backwards. A chunk used twice gets written twice, and Locate returns the first place. The end of a segment counts as inside it, because that’s where the cursor sits while you’re typing a word.

// Locate translates a position in the literate source to the first place
// tangling wrote it.
func (r *Result) Locate(line, col int) (o *Output, outLine, outCol int, ok bool) {
    for _, o := range r.Files {
        for n, l := range o.Lines {
            for _, s := range l.Segs {
                if s.SrcLine == line && col >= s.SrcCol && col <= s.SrcCol+s.Len {
                    return o, n, s.out(col), true
                }
            }
        }
    }
    return nil, 0, 0, false
}

That tool also proposes edits to the tangled file, and edits need more precision than error messages. Map sends a position in generated text to the nearest source line. Fine for a diagnostic. A disaster for a replacement. Exact only answers for text that actually came from the source. The end of a range is exclusive, so it can touch the end of a segment. An empty range is an insertion, and it can sit at either end of a segment.

// Exact is Map for the ends of an edit: it fails rather than approximate.
// The end of a range prefers the segment it closes, the start the one it opens.
func (o *Output) Exact(line, col int, end bool) (srcLine, srcCol int, ok bool) {
    if line < 0 || line >= len(o.Lines) {
        return 0, 0, false
    }
    for _, closing := range []bool{end, !end} {
        for _, s := range o.Lines[line].Segs {
            lo, hi := s.OutCol, s.OutCol+s.Len-1
            if closing {
                lo, hi = lo+1, hi+1
            }
            if col >= lo && col <= hi {
                return s.SrcLine, s.src(col), true
            }
        }
    }
    return 0, 0, false
}

Two halves of one map

Map and Locate read the same segments in opposite directions, and everything the editor does through gopls passes through both: a request goes out through Locate, and the answer comes back through Map. If they disagreed by one column, completion would insert text one character off and a rename would eat the letter next to the name. So they keep an invariant. Call a byte of the document that tangling wrote somewhere a tangled character, p, and a byte of an output file that a segment covers a written character, q. Then

\mathrm{Map}(\mathrm{Locate}(p)) = p

for every tangled character, no exceptions. The other direction needs one qualification, because tangling isn’t one-to-one. A chunk used twice is written twice, so two written characters share one source, and Locate can only return one of them. It returns the first, which makes

\mathrm{Locate}(\mathrm{Map}(q)) = q

true exactly when q is the first copy of its text. For a later copy it gives the first copy instead: a different place holding the same character from the same source. That’s the right question to ask gopls anyway, since a chunk is one piece of text no matter how many times it’s written.

In other words, Locate is a right inverse of Map everywhere, and a left inverse wherever tangling is one-to-one. Each function also accepts positions the other never produces, and the invariant says nothing about those:

Position Map Locate
a tangled or written character exact exact, the first copy
a generated line, added indentation the nearest segment never produced
prose, a reference, the markup of a directive never produced fails
the end of a segment the character before, or the chunk spliced in after the end of the segment

The last row is the one place where the two overlap and still disagree, and that’s on purpose. The end of a segment isn’t a character. It’s the gap after the last one, where the cursor sits while you type a word. Locate accepts it so completion works at the end of a line. Map has no use for gaps, because a compiler’s column always points at a character, so it sends a column past the end back to the last character. The function that inverts Locate there is Exact, asked for the end of a range.

What is proved, and what is tested

Each function does two things. It finds a segment, then it translates a column through it. The translation is arithmetic, it’s where an off-by-one would hide, and it’s the same in all three functions. So it’s written once, as two methods, and Map, Locate and Exact all call them. Their contracts are this section’s invariant, stated about a single segment, and litgo prove proves them:

// out is where the byte at column col of the source line went in the output.
//
// @ Requires s.SrcCol <= col ^ col <= s.SrcCol + s.Len
func (s Seg) out(col int) (c int) { return s.OutCol + col - s.SrcCol }

//@ Ensures s.OutCol <= c ^ c <= s.OutCol + s.Len
//@ Ensures s.src(c) = col

// src is where the byte at column col of the output line came from.
//
// @ Requires s.OutCol <= col ^ col <= s.OutCol + s.Len
func (s Seg) src(col int) (c int) { return s.SrcCol + col - s.OutCol }

//@ Ensures s.SrcCol <= c ^ c <= s.SrcCol + s.Len
//@ Ensures s.out(c) = col

(The // @ with a space is what gofmt makes of //@ inside a doc comment. The verifier reads both, and this spelling survives format-on-save.)

Read the second Ensures of each: src(out(col)) = col and out(src(col)) = col, for every segment there could ever be and every column in it or at its end. The first Ensures says the result is a column the other function accepts. That is \mathrm{Map} \circ \mathrm{Locate} and \mathrm{Locate} \circ \mathrm{Map} being the identity within a segment. Not a test that passed. A theorem.

Okay, but that’s not the whole invariant, and it would be wrong to pretend it was. Map(Locate(p)) = p also needs the search in Map to land on the segment the search in Locate started from. That holds because of how tangling lays segments out: on one output line they never overlap, and two segments from one source line are either the same text written twice or have a chunk reference between them. Those are facts about slices of structs holding slices, built by a recursive splice. The verifier’s world is integers and slices of integers, so it declines, and says so, instead of proving something about a simplified model that isn’t the program. And the bare claim “Map and Locate are inverses” is just false, as the table shows: Locate(Map(q)) = q fails for the second copy of a chunk, by design. So the labour splits like this. The arithmetic is proved, for all inputs. The search, and the layout it relies on, are tested in Map and Locate undo each other, character by character, on documents built to have every awkward case.

Editing structure

internal/lit/edit.go

Two edits change the shape of a document. litgo computes both of them, not the editor, so they can be tested, and so any editor can offer them. An edit replaces a range of lines. A list of edits runs from the bottom up, so you can apply them one after another and never adjust a line number.

// Edit replaces source lines [Start, End) with Lines. A list of edits is
// ordered bottom-up so an editor can apply them one after another.
type Edit struct {
    Start int      `json:"start"`
    End   int      `json:"end"`
    Lines []string `json:"lines"`
}

// Selection is a region of a document. With EndCol < 0 it is linewise;
// otherwise StartCol is the first selected byte and EndCol the byte just past
// the selection.
type Selection struct {
    StartLine, EndLine int // inclusive
    StartCol, EndCol   int
}

// UntangleResult describes the edits that extract a selection into a chunk.
type UntangleResult struct {
    Name    string `json:"name"`
    Inline  bool   `json:"inline"`
    RefLine int    `json:"ref_line"` // where the reference ends up
    DefLine int    `json:"def_line"` // first code line of the new chunk, after edits
    Edits   []Edit `json:"edits"`
}

Untangle turns a highlighted region into a named chunk. Reach for it when you are staring at a wall of code and think “this part should fold up and move to its own spot in the document”.

// Untangle moves the selection into a new named chunk at the bottom of the
// document and leaves a reference in its place. Because chunks are spliced in
// textually, the extracted code keeps the lexical scope it had.
func (d *Doc) Untangle(sel Selection, name string) (*UntangleResult, error) {
    if sel.StartLine < 0 || sel.EndLine >= len(d.Lines) || sel.StartLine > sel.EndLine {
        return nil, fmt.Errorf("selection is outside the document")
    }
    b := d.BlockAt(sel.StartLine)
    if b == nil || !b.Tangled() || sel.StartLine < b.First() || sel.EndLine > b.Last() {
        return nil, fmt.Errorf("select lines inside a single tangled code block")
    }
    comments := CommentsFor(b.Lang)
    if comments.Line == "" {
        return nil, fmt.Errorf("litgo does not know how to write a comment in %q, so it cannot leave a reference", b.Lang)
    }

    first, last := d.Lines[sel.StartLine], d.Lines[sel.EndLine]
    inline := sel.EndCol >= 0
    if inline {
        sel.StartCol = min(max(sel.StartCol, 0), len(first))
        sel.EndCol = min(max(sel.EndCol, 0), len(last))
        // A charwise selection that covers whole lines is really linewise.
        if strings.TrimSpace(first[:sel.StartCol]) == "" && strings.TrimSpace(last[sel.EndCol:]) == "" {
            inline = false
        }
        if inline && comments.Open == "" {
            return nil, fmt.Errorf("%s has no block comments, so only whole lines can be untangled", b.Lang)
        }
    }

    var body []string
    var replacement string
    if inline {
        hostIndent := first[:len(first)-len(strings.TrimLeft(first, " \t"))]
        if sel.StartLine == sel.EndLine {
            if sel.EndCol <= sel.StartCol {
                return nil, fmt.Errorf("selection is empty")
            }
            body = []string{first[sel.StartCol:sel.EndCol]}
        } else {
            body = append(body, first[sel.StartCol:])
            for i := sel.StartLine + 1; i < sel.EndLine; i++ {
                body = append(body, strings.TrimPrefix(d.Lines[i], hostIndent))
            }
            body = append(body, strings.TrimPrefix(last[:sel.EndCol], hostIndent))
        }
        if strings.TrimSpace(strings.Join(body, "")) == "" {
            return nil, fmt.Errorf("selection is empty")
        }
    } else {
        body = append(body, d.Lines[sel.StartLine:sel.EndLine+1]...)
        for len(body) > 0 && strings.TrimSpace(body[0]) == "" {
            body = body[1:]
            sel.StartLine++
        }
        for len(body) > 0 && strings.TrimSpace(body[len(body)-1]) == "" {
            body = body[:len(body)-1]
            sel.EndLine--
        }
        if len(body) == 0 {
            return nil, fmt.Errorf("selection is empty")
        }
    }

    name = NormName(name)
    if name == "" {
        name = SuggestName(body, inline)
    }
    name = d.uniqueName(name)

    if inline {
        ref := Ref{Inline: true, Open: comments.Open, Close: comments.Close}
        replacement = first[:sel.StartCol] + ref.Text(name) + last[sel.EndCol:]
    } else {
        indent := commonIndent(body)
        for i, l := range body {
            body[i] = strings.TrimPrefix(l, indent)
            if strings.TrimSpace(l) == "" {
                body[i] = ""
            }
        }
        replacement = indent + Ref{Open: comments.Line}.Text(name)
    }

    // Append the chunk at the bottom, separated by exactly one blank line.
    end := len(d.Lines)
    for end > 0 && strings.TrimSpace(d.Lines[end-1]) == "" {
        end--
    }
    tail := []string{"", "<!-- chunk: " + name + " -->", "```" + b.Lang}
    tail = append(tail, body...)
    tail = append(tail, "```")

    res := &UntangleResult{Name: name, Inline: inline, RefLine: sel.StartLine}
    res.Edits = []Edit{
        {Start: end, End: len(d.Lines), Lines: tail},
        {Start: sel.StartLine, End: sel.EndLine + 1, Lines: []string{replacement}},
    }
    res.DefLine = end + 3 - (sel.EndLine - sel.StartLine)
    return res, nil
}

func (d *Doc) uniqueName(name string) string {
    taken := map[string]bool{}
    for _, c := range d.Chunks() {
        taken[c.Name] = true
    }
    if !taken[name] {
        return name
    }
    for i := 2; ; i++ {
        if n := fmt.Sprintf("%s %d", name, i); !taken[n] {
            return n
        }
    }
}

func commonIndent(lines []string) string {
    indent, seen := "", false
    for _, l := range lines {
        if strings.TrimSpace(l) == "" {
            continue
        }
        ws := l[:len(l)-len(strings.TrimLeft(l, " \t"))]
        if !seen {
            indent, seen = ws, true
            continue
        }
        n := 0
        for n < len(indent) && n < len(ws) && indent[n] == ws[n] {
            n++
        }
        indent = indent[:n]
    }
    return indent
}

// Apply runs bottom-up edits against the document's lines.
func Apply(lines []string, edits []Edit) []string {
    out := append([]string{}, lines...)
    for _, e := range edits {
        next := append([]string{}, out[:e.Start]...)
        next = append(next, e.Lines...)
        out = append(next, out[e.End:]...)
    }
    return out
}

Rename changes a chunk’s name at every definition and every use. That takes the pressure off the heuristic that named the chunk. Its guess only has to be good enough to keep you moving, because a better name is one command away.

// ChunkAt names the chunk a line belongs to: a reference on the line, the
// chunk directive, or the chunk block around it. col < 0 means "anywhere".
func (d *Doc) ChunkAt(line, col int) string {
    if line < 0 || line >= len(d.Lines) {
        return ""
    }
    if b := d.BlockAt(line); b != nil && b.Tangled() {
        if line >= b.First() && line <= b.Last() {
            refs := d.refs(b, line)
            for _, r := range refs {
                if col >= r.Col && col < r.EndCol {
                    return r.Name
                }
            }
            if len(refs) > 0 {
                return refs[0].Name
            }
        }
        return b.Chunk
    }
    for _, b := range d.Blocks {
        if b.ChunkLine == line {
            return b.Chunk
        }
    }
    return ""
}

// Rename changes a chunk's name at every definition and every reference.
func (d *Doc) Rename(from, to string) ([]Edit, error) {
    from, to = NormName(from), NormName(to)
    if to == "" {
        return nil, fmt.Errorf("new chunk name is empty")
    }
    if strings.Contains(to, ">>") || strings.Contains(to, "-->") {
        return nil, fmt.Errorf("chunk names cannot contain >> or -->")
    }
    var target *Chunk
    for _, c := range d.Chunks() {
        if c.Name == to && from != to {
            return nil, fmt.Errorf("chunk ⟨%s⟩ already exists", to)
        }
        if c.Name == from {
            target = c
        }
    }
    if target == nil {
        return nil, fmt.Errorf("no chunk named ⟨%s⟩", from)
    }
    changed := map[int]string{}
    get := func(i int) string {
        if s, ok := changed[i]; ok {
            return s
        }
        return d.Lines[i]
    }
    for _, b := range target.Blocks {
        for _, dir := range d.Directives {
            if dir.Line == b.ChunkLine {
                l := d.Lines[dir.Line]
                changed[dir.Line] = l[:dir.ValueCol] + to + l[dir.ValueCol+len(dir.Value):]
            }
        }
    }
    // Rewrite references right-to-left so earlier columns stay valid.
    for i := len(target.Refs) - 1; i >= 0; i-- {
        r := target.Refs[i]
        l := get(r.Line)
        changed[r.Line] = l[:r.Col] + r.Text(to) + l[r.EndCol:]
    }
    var edits []Edit
    for i := len(d.Lines) - 1; i >= 0; i-- {
        if s, ok := changed[i]; ok {
            edits = append(edits, Edit{Start: i, End: i + 1, Lines: []string{s}})
        }
    }
    return edits, nil
}

An editor that speaks the Language Server Protocol asks its questions about a position: what is defined here, rename what is here. ChunkAt is generous and accepts any line in a chunk’s block, which is what a command wants. But inside a block, the thing under the cursor is Go, and Go tooling should answer for it. A chunk’s name is only mentioned in two places: in a reference, and in the directive that names it.

// Mention is a place where a chunk's name is written. Col and EndCol span
// the name itself.
type Mention struct {
    Chunk       *Chunk
    Ref         bool // in a reference, rather than in the chunk directive
    Line        int
    Col, EndCol int
}

// MentionAt finds the reference or the chunk directive at a position. A
// reference alone on its line owns the whole line.
func (d *Doc) MentionAt(line, col int) *Mention {
    if line < 0 || line >= len(d.Lines) {
        return nil
    }
    name, m := "", &Mention{Line: line}
    if b := d.BlockAt(line); b != nil {
        if !b.Tangled() || line < b.First() || line > b.Last() {
            return nil
        }
        for _, r := range d.refs(b, line) {
            if r.Inline && (col < r.Col || col >= r.EndCol) {
                continue
            }
            text := d.Lines[line][r.Col:r.EndCol]
            open, shut := strings.Index(text, "<<")+2, strings.LastIndex(text, ">>")
            inner := text[open:shut]
            m.Col = r.Col + open + len(inner) - len(strings.TrimLeft(inner, " \t"))
            m.EndCol = r.Col + shut - len(inner) + len(strings.TrimRight(inner, " \t"))
            name, m.Ref = r.Name, true
        }
    } else {
        for _, dir := range d.Directives {
            if dir.Line == line && dir.Key == "chunk" {
                name, m.Col, m.EndCol = NormName(dir.Value), dir.ValueCol, dir.ValueCol+len(dir.Value)
            }
        }
    }
    for _, c := range d.Chunks() {
        if c.Name == name {
            m.Chunk = c
            return m
        }
    }
    return nil
}

The other edit a language server asks for is a new import, when it completes a name from a package the file doesn’t import yet. The server proposes a new line for the import ( block. Problem: the document has no such block. litgo generates it from the imports: directive, so the directive is what gets the new entry.

// AddImport says what to insert, and where, for a file to import one more
// package. It fails for a file without a package directive, which writes
// its own import block.
func (d *Doc) AddImport(f *File, alias, path string) (line, col int, text string, ok bool) {
    entry := strings.TrimSpace(alias + " " + path)
    if n := len(f.Imports); n > 0 {
        last := f.Imports[n-1]
        return last.Line, last.Col + last.Len, ", " + entry, true
    }
    if f.PkgDir == nil {
        return 0, 0, "", false
    }
    return f.PkgDir.Line + 1, 0, "<!-- imports: " + entry + " -->\n", true
}

Naming a chunk

internal/lit/naming.go

If the code starts with a comment, that comment is the author’s own summary, so it becomes the name. Otherwise the shape of the first statement decides. for job := range jobs becomes for each job in jobs, if err != nil becomes handle err, and a select is named after the channels in its cases.

// SuggestName proposes a chunk name for extracted code. A leading comment is
// the author's own summary, so it wins; otherwise the first statement's shape
// decides. Names read as short phrases: they label intent in the host block.
func SuggestName(code []string, inline bool) string {
    var lines []string
    for _, l := range code {
        if t := strings.TrimSpace(l); t != "" {
            lines = append(lines, t)
        }
    }
    if len(lines) == 0 {
        return "chunk"
    }
    if !inline {
        if n := fromComment(lines); n != "" {
            return clip(n)
        }
    }
    for _, l := range lines {
        if _, ok := commentText(l); ok {
            continue
        }
        var n string
        if inline {
            n = fromExpr(l)
        } else {
            n = fromStmt(l, lines)
        }
        if n != "" {
            return clip(n)
        }
        break
    }
    if inline {
        return "expression"
    }
    return "chunk"
}

// commentText returns the words of a line comment in any of the comment
// syntaxes litgo knows.
func commentText(l string) (string, bool) {
    for _, leader := range []string{"//", "--", "#"} {
        if strings.HasPrefix(l, leader) && !strings.HasPrefix(l, "--[[") && !strings.HasPrefix(l, "#!") {
            return strings.TrimSpace(strings.TrimLeft(l, leader)), true
        }
    }
    return "", false
}

func fromComment(lines []string) string {
    text, ok := commentText(lines[0])
    if !ok || RefLineRe.MatchString(lines[0]) {
        return ""
    }
    if i := strings.IndexAny(text, ".;:!?"); i > 0 {
        text = text[:i]
    }
    words := strings.Fields(text)
    if len(words) == 0 || len(text) < 3 {
        return ""
    }
    if len(words) > 7 {
        words = words[:7]
    }
    // Lower-case a sentence-style first word, but leave identifiers
    // (HTTPServer, wg) alone.
    if w := []rune(words[0]); len(w) > 1 && unicode.IsUpper(w[0]) && unicode.IsLower(w[1]) && strings.ToLower(words[0][1:]) == words[0][1:] {
        w[0] = unicode.ToLower(w[0])
        words[0] = string(w)
    }
    return strings.Trim(strings.Join(words, " "), " ,-")
}
var (
    forRangeRe = regexp.MustCompile(`^for\s+(?:(.+?)\s*:?=\s*)?range\s+(.+?)\s*\{?$`)
    forCountRe = regexp.MustCompile(`^for\s+(\w+)\s*:=[^;]*;\s*\w+\s*[<>!]=?\s*([^;]+?)\s*;`)
    forCondRe  = regexp.MustCompile(`^for\s+(.+?)\s*\{?$`)
    ifRe       = regexp.MustCompile(`^if\s+(?:.*;\s*)?(.+?)\s*\{?$`)
    switchRe   = regexp.MustCompile(`^switch\s+(?:.*;\s*)?(.*?)\s*\{?$`)
    caseRecvRe = regexp.MustCompile(`^case\s+.*<-\s*([\w.]+(?:\(\))?)`)
    caseSendRe = regexp.MustCompile(`^case\s+([\w.]+)\s*<-`)
    funcRe     = regexp.MustCompile(`^func\s+(?:\([^)]*\)\s*)?(\w+)`)
    luaFuncRe  = regexp.MustCompile(`^(?:local\s+)?function\s+([\w.:]+)`)
    luaLocalRe = regexp.MustCompile(`^local\s+([\w, ]+?)\s*=`)
    declRe     = regexp.MustCompile(`^(type|var|const)\s+(\w+)`)
    assignRe   = regexp.MustCompile(`^([\w.\[\]*, ]+?)\s*(:=|=|\+=|-=)\s*(.+)$`)
    calleeRe   = regexp.MustCompile(`^&?\*?((?:[A-Za-z_][\w]*\.)*[A-Za-z_]\w*)\s*[({\[]`)
    sendRe     = regexp.MustCompile(`^([\w.]+)\s*<-\s*\S`)
    recvRe     = regexp.MustCompile(`^<-\s*([\w.]+)`)
)

func fromStmt(l string, all []string) string {
    switch {
    case strings.HasPrefix(l, "for"):
        if l == "for {" || l == "for" {
            return "loop forever"
        }
        if m := forRangeRe.FindStringSubmatch(l); m != nil {
            vars := strings.Split(m[1], ",")
            v := strings.TrimSpace(vars[len(vars)-1])
            if v == "" || v == "_" {
                return "loop over " + m[2]
            }
            return "for each " + v + " in " + m[2]
        }
        if m := forCountRe.FindStringSubmatch(l); m != nil {
            return "loop " + m[1] + " to " + m[2]
        }
        if m := forCondRe.FindStringSubmatch(l); m != nil {
            return "loop while " + m[1]
        }
    case strings.HasPrefix(l, "if "):
        if m := ifRe.FindStringSubmatch(l); m != nil {
            cond := m[1]
            if em := regexp.MustCompile(`^(\w*[eE]rr\w*)\s*!=\s*nil$`).FindStringSubmatch(cond); em != nil {
                return "handle " + em[1]
            }
            return "if " + cond
        }
    case strings.HasPrefix(l, "select"):
        var chans []string
        for _, c := range all {
            if m := caseRecvRe.FindStringSubmatch(c); m != nil {
                chans = append(chans, m[1])
            } else if m := caseSendRe.FindStringSubmatch(c); m != nil {
                chans = append(chans, m[1])
            }
        }
        if len(chans) > 3 {
            chans = chans[:3]
        }
        if len(chans) > 0 {
            return "select on " + strings.Join(chans, ", ")
        }
        return "select"
    case strings.HasPrefix(l, "switch"):
        if m := switchRe.FindStringSubmatch(l); m != nil && m[1] != "" {
            if subject, ok := strings.CutSuffix(m[1], ".(type)"); ok {
                if _, after, found := strings.Cut(subject, ":="); found {
                    subject = after
                }
                return "type switch on " + strings.TrimSpace(subject)
            }
            return "switch on " + m[1]
        }
        return "switch"
    case strings.HasPrefix(l, "defer "):
        rest := strings.TrimPrefix(l, "defer ")
        if strings.HasPrefix(rest, "func") {
            return "deferred cleanup"
        }
        if m := calleeRe.FindStringSubmatch(rest); m != nil {
            return "defer " + m[1]
        }
    case strings.HasPrefix(l, "go "):
        rest := strings.TrimPrefix(l, "go ")
        if strings.HasPrefix(rest, "func") {
            return "spawn goroutine"
        }
        if m := calleeRe.FindStringSubmatch(rest); m != nil {
            return "spawn " + m[1]
        }
    case strings.HasPrefix(l, "return"):
        rest := strings.TrimSpace(strings.TrimPrefix(l, "return"))
        if rest == "" {
            return "return"
        }
        return "return " + rest
    case strings.HasPrefix(l, "func"):
        if m := funcRe.FindStringSubmatch(l); m != nil {
            return "func " + m[1]
        }
    }
    if m := declRe.FindStringSubmatch(l); m != nil {
        return m[1] + " " + m[2]
    }
    if m := luaFuncRe.FindStringSubmatch(l); m != nil {
        return "function " + m[1]
    }
    if m := luaLocalRe.FindStringSubmatch(l); m != nil {
        return "define " + m[1]
    }
    if m := sendRe.FindStringSubmatch(l); m != nil {
        return "send to " + m[1]
    }
    if m := recvRe.FindStringSubmatch(l); m != nil {
        return "receive from " + m[1]
    }
    if m := assignRe.FindStringSubmatch(l); m != nil {
        var names []string
        for _, n := range strings.Split(m[1], ",") {
            n = strings.TrimSpace(n)
            if n != "_" && n != "err" && n != "ok" && n != "" {
                names = append(names, n)
            }
        }
        rhs := strings.TrimSpace(m[3])
        callee := ""
        if cm := calleeRe.FindStringSubmatch(rhs); cm != nil {
            callee = cm[1]
        }
        switch {
        case len(names) == 0 && callee != "":
            return "call " + callee
        case len(names) == 0:
            return "assign " + strings.TrimSpace(m[1])
        case callee == "make" || callee == "new":
            return callee + " " + strings.Join(names, ", ")
        case strings.HasPrefix(rhs, "func"):
            return "define " + strings.Join(names, ", ")
        case callee != "":
            return strings.Join(names, ", ") + " from " + callee
        case m[2] == ":=":
            return "define " + strings.Join(names, ", ")
        default:
            return "set " + strings.Join(names, ", ")
        }
    }
    if m := calleeRe.FindStringSubmatch(l); m != nil {
        return "call " + m[1]
    }
    return ""
}

func fromExpr(l string) string {
    switch {
    case strings.HasPrefix(l, "func"):
        return "func literal"
    case strings.HasPrefix(l, `"`) || strings.HasPrefix(l, "`"):
        return "string literal"
    }
    if m := calleeRe.FindStringSubmatch(l); m != nil {
        open := l[len(m[0])-1]
        switch {
        case open == '{':
            return m[1] + " literal"
        case open == '[':
            return m[1] + " element"
        case strings.HasSuffix(l, ")") && strings.Count(l, "(") == strings.Count(l, ")"):
            return m[1] + " call"
        }
        return m[1] + " expression"
    }
    if words := regexp.MustCompile(`[A-Za-z_][\w.]*`).FindAllString(l, 3); len(words) > 0 {
        return strings.Join(words, " ") + " expression"
    }
    return ""
}

// clip keeps names short and free of characters that would break the
// reference syntax or the HTML comment that carries the definition.
func clip(s string) string {
    s = strings.NewReplacer(">>", "> >", "<<", "< <", "-->", "->", "*/", "* /").Replace(s)
    s = strings.TrimRight(NormName(s), " {(")
    const limit = 48
    if r := []rune(s); len(r) > limit {
        s = string(r[:limit])
        if cut := strings.LastIndex(s, " "); cut > len(s)/2 {
            s = s[:cut]
        }
        s = strings.TrimRight(s, " ,.(&|") + "…"
    }
    return s
}

What the editor asks

internal/lit/analyze.go

After every change, the editor sends the buffer to litgo and gets back one answer with everything it needs: what to draw over the source, where the chunks are, and what is wrong.

// Item is something an editor can render in place of, or next to, source text.
type Item struct {
    Kind    string   `json:"kind"` // math, math-block, mermaid, table, chunk-def, chunk-ref, file, vego, vego-math
    Line    int      `json:"line"`
    Col     int      `json:"col"`
    EndLine int      `json:"end_line"`
    EndCol  int      `json:"end_col"`
    Text    string   `json:"text,omitempty"`  // one-line replacement
    Lines   []string `json:"lines,omitempty"` // block rendering
    Info    string   `json:"info,omitempty"`  // end-of-line annotation
    Name    string   `json:"name,omitempty"`
    Error   string   `json:"error,omitempty"`

    Table *Table `json:"table,omitempty"`
}

// ChunkInfo locates a chunk's definitions and uses for navigation.
type ChunkInfo struct {
    Name string     `json:"name"`
    Defs []ChunkDef `json:"defs"`
    Refs []ChunkUse `json:"refs"`
}

type ChunkDef struct {
    Directive int `json:"directive"`
    First     int `json:"first"` // first code line
    Last      int `json:"last"`  // last code line, inclusive
}

type ChunkUse struct {
    Line   int `json:"line"`
    Col    int `json:"col"`
    EndCol int `json:"end_col"`
}

// Analysis is everything the editor needs after a change.
type Analysis struct {
    Items       []Item      `json:"items"`
    Chunks      []ChunkInfo `json:"chunks"`
    Diagnostics []Diag      `json:"diagnostics"`
    Outputs     []string    `json:"outputs"`
}

// AnalyzeOptions tune rendering.
type AnalyzeOptions struct {
    Upright bool
}
// Analyze renders math and diagrams, annotates chunks, and checks the
// document: undefined or unused chunks, cycles, and Go syntax errors.
func (d *Doc) Analyze(o AnalyzeOptions) *Analysis {
    a := &Analysis{Items: []Item{}, Chunks: []ChunkInfo{}, Diagnostics: []Diag{}, Outputs: []string{}}
    res := d.Tangle()
    for _, o := range res.Files {
        a.Outputs = append(a.Outputs, o.Path)
    }
    a.Diagnostics = append(a.Diagnostics, res.Diags...)
    a.Diagnostics = append(a.Diagnostics, res.SyntaxCheck()...)

    inBlock := make([]bool, len(d.Lines))
    for _, b := range d.Blocks {
        for i := b.Open; i <= b.Close && i < len(d.Lines); i++ {
            inBlock[i] = true
        }
        switch b.Lang {
        case "mermaid":
            it := Item{Kind: "mermaid", Line: b.Open, EndLine: min(b.Close, len(d.Lines)-1)}
            lines, err := mermaid.Render(strings.Join(d.Lines[b.First():b.Last()+1], "\n"))
            if err != nil {
                it.Error = err.Error()
                it.Lines = []string{"[mermaid: " + err.Error() + "]"}
            } else {
                it.Lines = lines
            }
            a.Items = append(a.Items, it)
        case "math", "latex", "tex":
            if b.Last() >= b.First() {
                src := strings.Join(d.Lines[b.First():b.Last()+1], "\n")
                a.Items = append(a.Items, Item{Kind: "math-block", Line: b.Open, EndLine: min(b.Close, len(d.Lines)-1),
                    Lines: latex.Render(src, latex.Options{Display: true, Upright: o.Upright})})
            }
        case "go":
            a.Items = append(a.Items, d.proofItems(b, o)...)
        }
    }
    a.Items = append(a.Items, d.mathItems(inBlock, o)...)
    a.Items = append(a.Items, d.tableItems(inBlock, o)...)

    for _, f := range d.Files[1:] {
        a.Items = append(a.Items, Item{Kind: "file", Line: f.Line, EndLine: f.Line, EndCol: len(d.Lines[f.Line]), Text: "▸ " + f.Path})
    }

    chunks := d.Chunks()
    for _, c := range chunks {
        info := ChunkInfo{Name: c.Name, Defs: []ChunkDef{}, Refs: []ChunkUse{}}
        for _, b := range c.Blocks {
            info.Defs = append(info.Defs, ChunkDef{Directive: b.ChunkLine, First: b.First(), Last: b.Last()})
        }
        for _, r := range c.Refs {
            info.Refs = append(info.Refs, ChunkUse{Line: r.Line, Col: r.Col, EndCol: r.EndCol})
        }
        a.Chunks = append(a.Chunks, info)
    }
    for _, c := range chunks {
        for i, b := range c.Blocks {
            text := "⟨" + c.Name + "⟩ ≡"
            if i > 0 {
                text = "⟨" + c.Name + "⟩ +≡"
            }
            info := "unused"
            switch n := len(c.Refs); {
            case n == 1:
                info = fmt.Sprintf("← used at line %d", c.Refs[0].Line+1)
            case n > 1:
                var at []string
                for _, r := range c.Refs {
                    at = append(at, fmt.Sprint(r.Line+1))
                }
                info = "← used at lines " + strings.Join(at, ", ")
            }
            a.Items = append(a.Items, Item{Kind: "chunk-def", Name: c.Name, Line: b.ChunkLine, EndLine: b.ChunkLine,
                Col: 0, EndCol: len(d.Lines[b.ChunkLine]), Text: text, Info: info})
        }
        for _, r := range c.Refs {
            it := Item{Kind: "chunk-ref", Name: c.Name, Line: r.Line, EndLine: r.Line, Col: r.Col, EndCol: r.EndCol, Text: "⟨" + c.Name + "⟩"}
            if len(c.Blocks) == 0 {
                it.Info = "undefined"
            } else {
                n := 0
                for _, b := range c.Blocks {
                    n += b.Last() - b.First() + 1
                }
                it.Info = fmt.Sprintf("→ line %d · %d line%s", c.Blocks[0].First()+1, n, plural(n))
            }
            if r.Inline {
                it.Info = ""
            }
            a.Items = append(a.Items, it)
        }
    }
    return a
}

func plural(n int) string {
    if n == 1 {
        return ""
    }
    return "s"
}

Finding the annotations

A //@ comment is a specification (Proofs has the whole story). To a syntax highlighter it’s just another comment, grey and easy to skip. But it is the most carefully chosen line in the block, so the editor is told where the annotations are. The item runs from the //@ to the end of the line, and Name is the keyword, which an editor can set apart from the formula.

// ProofKeywords are the words an annotation can start with. One that starts
// with none of them is an Assert.
var ProofKeywords = map[string]bool{"Requires": true, "Ensures": true, "Exsures": true, "Invariant": true,
    "Variant": true, "Measure": true, "BaseCase": true, "InductionHypothesis": true, "Assert": true,
    "Axiom": true, "Preserves": true, "Predicate": true, "Immutable": true, "Property": true}

// ProofComment reports whether a line comment is an annotation, and returns
// what follows the marker. The marker is //@. gofmt rewrites that to // @ in
// a doc comment, so that is accepted too, but only in front of a keyword or
// where the annotation above is not finished (continuing), because a comment
// may well start with an @ for reasons of its own.
func ProofComment(text string, continuing bool) (body string, ok bool) {
    if strings.HasPrefix(text, "//@") {
        return text[3:], true
    }
    if !strings.HasPrefix(text, "// @") {
        return "", false
    }
    body = text[4:]
    word, _, _ := strings.Cut(strings.TrimLeft(body, " \t"), " ")
    return body, continuing || ProofKeywords[word]
}

// proofItems finds the VeGo annotations of a Go block: one item for the
// annotation, and one for its formula, typeset.
func (d *Doc) proofItems(b *Block, o AnalyzeOptions) []Item {
    var out []Item
    continuing := false
    for i := b.First(); i <= b.Last() && i < len(d.Lines); i++ {
        line := d.Lines[i]
        text := strings.TrimLeft(line, " \t")
        body, ok := ProofComment(text, continuing)
        if !ok {
            continuing = false
            continue
        }
        word, _, _ := strings.Cut(strings.TrimLeft(body, " \t"), " ")
        if continuing || !ProofKeywords[word] {
            word = "" // a formula by itself is an Assert without the word
        }
        trimmed := strings.TrimRight(body, " \t")
        continuing = strings.HasSuffix(trimmed, "||") || strings.HasSuffix(trimmed, " v") || strings.HasSuffix(trimmed, " in") ||
            trimmed != "" && strings.ContainsAny(trimmed[len(trimmed)-1:], "^:=<>+-*/%(,[~!&")
        out = append(out, Item{Kind: "vego", Line: i, EndLine: i, Col: len(line) - len(text), EndCol: len(line), Name: word})
        formula := strings.TrimLeft(body, " \t")
        if word != "" {
            formula = strings.TrimLeft(formula[len(word):], " \t")
        }
        if math := proofMath(formula, o); math != "" {
            out = append(out, Item{Kind: "vego-math", Line: i, EndLine: i, Col: len(line) - len(formula),
                EndCol: len(strings.TrimRight(line, " \t")), Text: math})
        }
    }
    return out
}

A formula is ASCII so it can be typed, and mathematics so it can be read, and those two pull in opposite directions. lo*lo <= n ^ n < hi*hi is what the keyboard has. lo^2 \leq n \land n < hi^2 is what it means. The prose already gets this treatment: $...$ is drawn as Unicode and turns back into its source under the cursor. So a formula is translated into the LaTeX it would have been in the prose, and handed to the same renderer. It comes out looking like the math around it, italic letters and all, and nothing here knows what a ≤ is.

The translation is a scan for words and operators. It doesn’t parse, because it has to be right about a formula that is only half typed. The one tricky letter is v, which means or between two operands and is a variable anywhere else. If the renderer leaves a backslash in its answer, it met something it didn’t know, and the raw source beats a guess.

// proofOps are the operators of a formula, longest first, and their LaTeX.
var proofOps = []struct{ op, tex string }{
    {"<->", `\leftrightarrow `}, {"::=", `\coloneqq `}, {"...", `\infty `}, {"->", `\to `}, {"<-", `\leftarrow `},
    {"<=", `\leq `}, {">=", `\geq `}, {"<>", `\neq `}, {"!=", `\neq `}, {"==", `= `}, {"&&", `\land `}, {"||", `\lor `},
    {"^", `\land `}, {"~", `\lnot `}, {"!", `\lnot `}, {"*", `\cdot `}, {"%", `\bmod `},
}

var proofWords = map[string]string{"Forall": `\forall `, "Exists": `\exists `, "Unique": `\exists! `, "in": `\in `,
    "true": `\text{true}`, "false": `\text{false}`, "len": `\text{len}`}

// proofPower counts the factors of a product of one name with itself,
// x*x*x, where the first x ends at j, and says where the product ends. A
// name that is called or indexed is not the same factor.
func proofPower(formula string, j int, word string) (n, end int) {
    n, end = 1, j
    for {
        k := end
        for k < len(formula) && formula[k] == ' ' {
            k++
        }
        if k >= len(formula) || formula[k] != '*' {
            return n, end
        }
        for k++; k < len(formula) && formula[k] == ' '; k++ {
        }
        if !strings.HasPrefix(formula[k:], word) {
            return n, end
        }
        k += len(word)
        if k < len(formula) {
            if c := formula[k]; c == '_' || c == '.' || c == '\'' || c == '(' || c == '[' ||
                c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' {
                return n, end
            }
        }
        n, end = n+1, k
    }
}

// proofClose finds the ) that closes the ( at i, or returns 0.
func proofClose(formula string, i int) int {
    depth := 0
    for k := i; k < len(formula); k++ {
        switch formula[k] {
        case '(':
            depth++
        case ')':
            if depth--; depth == 0 {
                return k
            }
        }
    }
    return 0
}

// proofMath typesets a formula the way the math in the prose is typeset.
func proofMath(formula string, o AnalyzeOptions) string {
    var tex strings.Builder
    operand := false           // was the last thing something an operator can follow
    quantifiers := 0           // that still wait for the colon before their body
    powers := map[int][2]int{} // by the ) of a repeated factor: how often, and where the product ends
    last := ""
    for i := 0; i < len(formula); {
        c := formula[i]
        switch {
        case c == ' ' || c == '\t':
            i++
        case c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z':
            j := i
            for j < len(formula) && (formula[j] == '_' || formula[j] == '.' && j+1 < len(formula) && formula[j+1] != '.' ||
                formula[j] >= 'a' && formula[j] <= 'z' || formula[j] >= 'A' && formula[j] <= 'Z' || formula[j] >= '0' && formula[j] <= '9') {
                j++
            }
            word := formula[i:j]
            for j < len(formula) && formula[j] == '\'' {
                j++
            }
            primes := formula[i+len(word) : j]
            switch {
            case word == "v" && primes == "" && operand:
                tex.WriteString(`\lor `)
                operand = false
            case word == "Z" && last == "in":
                tex.WriteString(`\mathbb{Z}`)
                operand = true
            case proofWords[word] != "" && primes == "":
                if word == "Forall" || word == "Exists" || word == "Unique" {
                    quantifiers++
                }
                tex.WriteString(proofWords[word])
                operand = word == "true" || word == "false"
            default:
                tex.WriteString(strings.ReplaceAll(word, "_", `\_`) + primes)
                // x*x is a square, and is read more easily as one. After a / or a %
                // it is not: a/x*x is (a/x)*x.
                if n, end := proofPower(formula, j, word); n > 1 && primes == "" && last != "/" && last != "%" {
                    fmt.Fprintf(&tex, "^{%d}", n)
                    j = end
                }
                tex.WriteString(" ")
                operand = true
            }
            last = word
            i = j
        default:
            found := false
            for _, op := range proofOps {
                if strings.HasPrefix(formula[i:], op.op) {
                    tex.WriteString(op.tex)
                    i += len(op.op)
                    found, operand, last = true, op.op == "...", op.op
                    break
                }
            }
            if !found && c == ':' && quantifiers > 0 {
                // The colon of a quantifier gets room; the one in A[a:b) does not.
                quantifiers--
                tex.WriteString(`\;:\;`)
                found, operand, last = true, false, ":"
                i++
            }
            if c == '(' && !operand && last != "/" && last != "%" {
                // (r+1)*(r+1) is a square too. Remember it until its first ) comes by.
                if close := proofClose(formula, i); close > 0 {
                    if n, end := proofPower(formula, close+1, formula[i:close+1]); n > 1 {
                        powers[close] = [2]int{n, end}
                    }
                }
            }
            if pw, ok := powers[i]; ok && c == ')' {
                fmt.Fprintf(&tex, ")^{%d}", pw[0])
                found, operand, last = true, true, ")"
                i = pw[1]
            }
            if !found {
                tex.WriteByte(c)
                operand = c == ')' || c == ']' || c == '|' || c >= '0' && c <= '9'
                last = string(c)
                i++
            }
        }
    }
    if tex.Len() == 0 {
        return ""
    }
    out := latex.Render(tex.String(), latex.Options{Upright: o.Upright})
    if len(out) != 1 || strings.Contains(out[0], `\`) {
        return ""
    }
    return out[0]
}

Finding the math

Inline math follows pandoc’s rules. The opening $ has to touch the formula on its right, and the closing $ has to touch it on its left and can’t be followed by a digit. That way a sentence about $5 and $10 stays a sentence.

var displayOpenRe = regexp.MustCompile(`^\s*(\$\$|\\\[)`)

// mathItems finds LaTeX in prose: $inline$, \(inline\), and display math in
// $$...$$ or \[...\], on one line or spanning several.
func (d *Doc) mathItems(inBlock []bool, o AnalyzeOptions) []Item {
    var items []Item
    for i := 0; i < len(d.Lines); i++ {
        if inBlock[i] || directiveRe.MatchString(d.Lines[i]) {
            continue
        }
        line := d.Lines[i]
        if m := displayOpenRe.FindStringSubmatch(line); m != nil {
            closer := map[string]string{"$$": "$$", `\[`: `\]`}[m[1]]
            body := strings.TrimSpace(line)[len(m[1]):]
            end := -1
            var src []string
            if rest := strings.TrimSpace(body); strings.HasSuffix(rest, closer) && len(rest) >= len(closer) {
                end, src = i, []string{strings.TrimSuffix(rest, closer)}
            } else if !strings.Contains(body, closer) {
                src = []string{body}
                for j := i + 1; j < len(d.Lines) && j < i+200 && !inBlock[j]; j++ {
                    t := strings.TrimSpace(d.Lines[j])
                    if strings.HasSuffix(t, closer) {
                        src = append(src, strings.TrimSuffix(t, closer))
                        end = j
                        break
                    }
                    src = append(src, d.Lines[j])
                }
            }
            if end >= 0 {
                if tex := strings.TrimSpace(strings.Join(src, "\n")); tex != "" {
                    items = append(items, Item{Kind: "math-block", Line: i, EndLine: end, EndCol: len(d.Lines[end]),
                        Lines: latex.Render(tex, latex.Options{Display: true, Upright: o.Upright})})
                }
                i = end
                continue
            }
        }
        items = append(items, inlineMath(line, i, o)...)
    }
    return items
}

func inlineMath(line string, lineNo int, o AnalyzeOptions) []Item {
    var items []Item
    add := func(start, end int, tex string) {
        tex = strings.TrimSpace(tex)
        if tex == "" {
            return
        }
        text := latex.Render(tex, latex.Options{Upright: o.Upright})[0]
        items = append(items, Item{Kind: "math", Line: lineNo, EndLine: lineNo, Col: start, EndCol: end, Text: text})
    }
    for i := 0; i < len(line); i++ {
        switch c := line[i]; {
        case c == '\\' && i+1 < len(line) && line[i+1] == '(':
            if j := strings.Index(line[i+2:], `\)`); j >= 0 {
                add(i, i+2+j+2, line[i+2:i+2+j])
                i += 2 + j + 1
            }
        case c == '\\':
            i++ // escaped character, including \$
        case c == '`':
            // Skip the code span: a run of n backticks closes at the next run of n.
            n := 1
            for i+n < len(line) && line[i+n] == '`' {
                n++
            }
            if j := strings.Index(line[i+n:], strings.Repeat("`", n)); j >= 0 {
                i += n + j + n - 1
            } else {
                i += n - 1
            }
        case c == '$':
            delim := "$"
            if i+1 < len(line) && line[i+1] == '$' {
                delim = "$$"
            }
            start := i + len(delim)
            if start >= len(line) || line[start] == ' ' || line[start] == '\t' {
                i = start - 1
                continue
            }
            // The closer must hug the formula and not be followed by a digit,
            // so prices like $5 and $10 stay prose.
            for j := start; j < len(line); j++ {
                if line[j] == '\\' {
                    j++
                    continue
                }
                if !strings.HasPrefix(line[j:], delim) {
                    continue
                }
                after := j + len(delim)
                if line[j-1] == ' ' && after < len(line) && line[after] != ' ' {
                    break // this $ opens something itself, so ours was a literal dollar
                }
                if line[j-1] == ' ' || (delim == "$" && after < len(line) && line[after] >= '0' && line[after] <= '9') {
                    continue
                }
                add(i, after, line[start:j])
                i = after - 1
                break
            }
        }
    }
    return items
}

Finding the tables

A table is a header row, then a rule like |---|:-:|, then the rows. When the cells have different lengths, the pipes zigzag and the source is hard to read. The editor straightens them in place. Every line stays where it is, and spaces go in front of the pipes until they line up.

Here’s the catch: to do that, the editor needs to know how wide each cell looks, and that isn’t how wide its source is. Math is drawn as Unicode, and the backticks around code, the asterisks around emphasis and the address of a link are all hidden. So for each cell litgo reports where it sits in the line and what its text looks like after all of that. It also lists the pieces of markup to hide, so the editor hides exactly what was left out of the text, whether or not it would have hidden it anyway. The colons in the rule say how each column is aligned.

// Table is what an editor needs to line up the columns of a table in place.
type Table struct {
    Align []string   `json:"align"` // left, right or center, for each column
    Rule  []Cell     `json:"rule"`  // the cells of the rule under the header
    Rows  []TableRow `json:"rows"`  // the header, and then the rows under the rule
}

type TableRow struct {
    Line  int      `json:"line"`
    Cells []Cell   `json:"cells"`
    Hide  [][2]int `json:"hide,omitempty"` // columns of the markup that is not shown
}

// Cell is one cell of a row. From and To are the pipes around it (or the
// ends of the row), Col and EndCol are its text without the spaces around it.
type Cell struct {
    From   int    `json:"from"`
    To     int    `json:"to"`
    Col    int    `json:"col"`
    EndCol int    `json:"end_col"`
    Text   string `json:"text"` // what the reader sees
}

var tableRuleRe = regexp.MustCompile(`^\s*\|?\s*:?-+:?\s*(\|\s*:?-+:?\s*)*\|?\s*$`)

// tableItems finds the tables in prose.
func (d *Doc) tableItems(inBlock []bool, o AnalyzeOptions) []Item {
    var items []Item
    row := func(i int) bool {
        return i < len(d.Lines) && !inBlock[i] && strings.Contains(d.Lines[i], "|") && !directiveRe.MatchString(d.Lines[i])
    }
    for i := 0; i < len(d.Lines); i++ {
        if !row(i) || !row(i+1) || !tableRuleRe.MatchString(d.Lines[i+1]) {
            continue
        }
        t := &Table{Rule: cells(d.Lines[i+1])}
        if len(cells(d.Lines[i])) != len(t.Rule) {
            continue
        }
        for _, r := range t.Rule {
            rule := d.Lines[i+1][r.Col:r.EndCol]
            switch left, right := strings.HasPrefix(rule, ":"), strings.HasSuffix(rule, ":"); {
            case left && right:
                t.Align = append(t.Align, "center")
            case right:
                t.Align = append(t.Align, "right")
            default:
                t.Align = append(t.Align, "left")
            }
        }
        end := i + 1
        for j := i; row(j); j++ {
            if j == i+1 {
                continue
            }
            r := TableRow{Line: j, Cells: cells(d.Lines[j])}
            for c := range r.Cells {
                cell := &r.Cells[c]
                var hide [][2]int
                cell.Text, hide = cellText(d.Lines[j], cell.Col, cell.EndCol, o)
                r.Hide = append(r.Hide, hide...)
            }
            t.Rows = append(t.Rows, r)
            end = j
        }
        items = append(items, Item{Kind: "table", Line: i, EndLine: end, EndCol: len(d.Lines[end]), Table: t})
        i = end
    }
    return items
}

// cells splits a row at its pipes. A pipe with a backslash in front of it is
// part of a cell, and the pipes at the ends of the row are not cells.
func cells(line string) []Cell {
    start, end := 0, len(line)
    for start < end && (line[start] == ' ' || line[start] == '\t') {
        start++
    }
    for end > start && (line[end-1] == ' ' || line[end-1] == '\t') {
        end--
    }
    if start < end && line[start] == '|' {
        start++
    }
    if end > start && line[end-1] == '|' && (end-1 == start || line[end-2] != '\\') {
        end--
    }
    var out []Cell
    from := start
    for i := start; i <= end; i++ {
        if i == end || line[i] == '|' && line[i-1] != '\\' {
            c := Cell{From: from, To: i, Col: from, EndCol: i}
            for c.Col < c.EndCol && line[c.Col] == ' ' {
                c.Col++
            }
            for c.EndCol > c.Col && line[c.EndCol-1] == ' ' {
                c.EndCol--
            }
            out = append(out, c)
            from = i + 1
        }
    }
    return out
}

Math is left to inlineMath, which the editor draws the same way in a table as anywhere else. Everything between the formulas gets searched for markup. Each kind of markup is one alternative in the pattern, with a group around the part that stays visible. Whatever is in the match but outside the group is hidden.

var markupRe = regexp.MustCompile("`+([^`]+)`+" + `|\*\*([^*]+)\*\*|\*([^*]+)\*|\[([^\]]+)\]\([^)]*\)|\\(\|)`)

// cellText is line[col:end] the way a reader sees it, and the columns of
// the markup that the reader does not see.
func cellText(line string, col, end int, o AnalyzeOptions) (string, [][2]int) {
    var sb strings.Builder
    var hide [][2]int
    plain := func(from, to int) {
        at := from
        for _, m := range markupRe.FindAllStringSubmatchIndex(line[from:to], -1) {
            for g := 2; g < len(m); g += 2 {
                if m[g] >= 0 {
                    sb.WriteString(line[at:from+m[0]] + line[from+m[g]:from+m[g+1]])
                    hide = append(hide, [2]int{from + m[0], from + m[g]})
                    if m[g+1] < m[1] {
                        hide = append(hide, [2]int{from + m[g+1], from + m[1]})
                    }
                }
            }
            at = from + m[1]
        }
        sb.WriteString(line[at:to])
    }
    at := col
    for _, m := range inlineMath(line[col:end], 0, o) {
        plain(at, col+m.Col)
        sb.WriteString(m.Text)
        at = col + m.EndCol
    }
    plain(at, end)
    return sb.String(), hide
}

Tests

The sample documents in these tests are full of reference syntax that is only data. The verbatim directive exists for exactly this.

internal/lit/lit_test.go
const sample = "<!-- package: main -->\n<!-- imports: fmt, os -->\n<!-- tangler-exclude: ^//.*$ -->\n<!-- tangler-exclude: ^<!-- .* -->$ -->\n\n# T\n\n```go\n// dropped\nfunc main() {\n\tfor i := 0; i < 3; i++ {\n\t\t// <<body>>\n\t}\n\tfmt.Println(add(/*<<args>>*/))\n}\n```\n\n<!-- chunk: body -->\n```go\n// Print the index.\nif i > 0 {\n\tfmt.Println(i)\n}\n```\n\n<!-- chunk: args -->\n```go\n1,\n2\n```\n\n```text\nnot code\n```\n"

// program is everything a document tangles to.
func program(d *Doc) string {
    var sb strings.Builder
    for _, o := range d.Tangle().Files {
        sb.WriteString("==> " + o.Path + "\n")
        sb.Write(o.Bytes())
    }
    return sb.String()
}

func TestDirectives(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    if f := d.Files[0]; f.Package != "main" || len(f.Imports) != 2 || f.Imports[1].Path != "os" {
        t.Fatalf("directives: %+v", d)
    }
    if len(d.Excludes) != 2 || d.Excludes[1].String() != "^<!-- .* -->$" {
        t.Fatalf("an exclude pattern containing --> must survive: %v", d.Excludes)
    }
    if got := d.OutPath(d.Files[0]); got != "x.go" {
        t.Errorf("OutPath = %q", got)
    }
}

func TestTangleAndMap(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    res := d.Tangle()
    if res.HasErrors() || len(res.Files) != 1 {
        t.Fatalf("diags: %+v", res.Diags)
    }
    r := res.Files[0]
    out := string(r.Bytes())
    for _, want := range []string{"\t\tif i > 0 {\n\t\t\tfmt.Println(i)\n\t\t}\n", "\tfmt.Println(add(1,\n\t2))\n", "import (\n\t\"fmt\"\n\t\"os\"\n)"} {
        if !strings.Contains(out, want) {
            t.Errorf("output lacks %q:\n%s", want, out)
        }
    }
    if strings.Contains(out, "// dropped") {
        t.Error("tangler-exclude did not drop the top-level comment")
    }
    if strings.Contains(out, "// Print the index.") {
        t.Error("excludes apply to source lines, so a chunk's column-0 comment is dropped too")
    }
    // Every mapped position must point at identical text in the source.
    for ln, l := range r.Lines {
        for _, s := range l.Segs {
            if s.OutCol == 0 && strings.HasPrefix(l.Text, "\t\"") || strings.HasPrefix(l.Text, "package ") {
                continue // generated header lines map to directive values
            }
            got := d.Lines[s.SrcLine][s.SrcCol : s.SrcCol+s.Len]
            if want := l.Text[s.OutCol : s.OutCol+s.Len]; got != want {
                t.Errorf("out line %d: segment maps %q to %q", ln, want, got)
            }
        }
    }
    // The `2` of the inline chunk sits on a continuation line.
    for ln, l := range r.Lines {
        if l.Text == "\t2))" {
            sl, sc, via, _ := r.Map(ln, 1)
            if d.Lines[sl] != "2" || sc != 0 || len(via) != 1 {
                t.Errorf("Map = %d:%d via %v", sl, sc, via)
            }
            if sl, _, _, _ := r.Map(ln, 3); !strings.Contains(d.Lines[sl], "add(") {
                t.Errorf("text after an inline chunk belongs to the host line, got %q", d.Lines[sl])
            }
        }
    }
    if diags := res.SyntaxCheck(); len(diags) != 0 {
        t.Errorf("syntax: %+v", diags)
    }
}

func TestLocateAndExact(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    res := d.Tangle()
    src := 0
    for i, l := range d.Lines {
        if l == "\tfmt.Println(i)" {
            src = i
        }
    }
    o, ol, oc, ok := res.Locate(src, 5)
    if !ok || !strings.HasPrefix(o.Lines[ol].Text[oc:], "Println(i)") {
        t.Fatalf("Locate = %d:%d %v", ol, oc, ok)
    }
    if sl, sc, ok := o.Exact(ol, oc, false); !ok || sl != src || sc != 5 {
        t.Errorf("Exact should undo Locate, got %d:%d %v", sl, sc, ok)
    }
    if _, sc, ok := o.Exact(ol, len(o.Lines[ol].Text), true); !ok || sc != len(d.Lines[src]) {
        t.Errorf("the end of a range may touch the end of the line, got %d %v", sc, ok)
    }
    if _, _, ok := o.Exact(ol, 0, false); ok {
        t.Error("indentation that tangling added came from nowhere")
    }
    if _, _, ok := o.Exact(4, 0, false); ok || o.Lines[4].Text != "import (" {
        t.Errorf("neither did %q", o.Lines[4].Text)
    }
    if _, _, _, ok := res.Locate(5, 0); ok {
        t.Error("prose is not written anywhere")
    }
}

Map and Locate undo each other

The tests above check Map and Locate at a few hand-picked positions. But the invariant from Two halves of one map makes a claim about every position, so these tests visit every one: each byte of each segment of each file, in three documents. sample has an inline chunk spliced into the middle of a line. multi tangles to several files in several languages. twice covers the one thing the others lack, a chunk written twice, at two different depths of indentation.

const twice = "<!-- package: main -->\n\n```go\nfunc a() {\n\t// <<step>>\n}\n\n" +
    "func b() {\n\tif true {\n\t\t// <<step>>\n\t}\n}\n```\n\n" +
    "<!-- chunk: step -->\n```go\nprintln(1)\n\nprintln(2)\n```\n"

var roundTrips = map[string]string{"sample": sample, "multi": multi, "twice": twice}

// char is one byte that tangling copied: where it was written, and where from.
type char struct {
    out               *Output
    line, col         int
    srcLine, srcCol   int
    first, segmentEnd bool // the first copy of its source; the last byte of its segment
}

// chars lists every written character of a result, in the order Locate
// searches: file by file, line by line.
func chars(res *Result) []char {
    var all []char
    seen := map[[2]int]bool{}
    for _, o := range res.Files {
        for n, l := range o.Lines {
            for _, s := range l.Segs {
                for k := 0; k < s.Len; k++ {
                    src := [2]int{s.SrcLine, s.SrcCol + k}
                    all = append(all, char{o, n, s.OutCol + k, src[0], src[1], !seen[src], k == s.Len-1})
                    seen[src] = true
                }
            }
        }
    }
    return all
}

From the source and back is the half with no exceptions. Wherever a tangled character was written, and however many times, Locate finds a place and Map returns from it to the very byte that was asked about. Every time.

func TestMapUndoesLocate(t *testing.T) {
    for name, src := range roundTrips {
        res := Parse("x.lit.md", []byte(src)).Tangle()
        all := chars(res)
        if len(all) == 0 {
            t.Fatalf("%s: nothing was tangled", name)
        }
        for _, c := range all {
            o, line, col, ok := res.Locate(c.srcLine, c.srcCol)
            if !ok {
                t.Errorf("%s: Locate(%d:%d) finds nothing, but it was written at %s:%d:%d", name, c.srcLine, c.srcCol, c.out.Path, c.line, c.col)
                continue
            }
            if sl, sc, _, ok := o.Map(line, col); !ok || sl != c.srcLine || sc != c.srcCol {
                t.Errorf("%s: Map(Locate(%d:%d)) = %d:%d %v", name, c.srcLine, c.srcCol, sl, sc, ok)
            }
        }
    }
}

From the output and back, Map is exact on every written character, and Locate returns to the same place if that place is the first copy. From a later copy it returns to the first one, and the test pins down what “the first one” means: an earlier position whose source is the same byte. The test also checks that twice really has later copies and the other two documents really don’t. Otherwise that branch could pass by never running at all.

func TestLocateUndoesMap(t *testing.T) {
    for name, src := range roundTrips {
        res := Parse("x.lit.md", []byte(src)).Tangle()
        copies := 0
        for _, c := range chars(res) {
            sl, sc, _, ok := c.out.Map(c.line, c.col)
            if !ok || sl != c.srcLine || sc != c.srcCol {
                t.Errorf("%s: Map(%s:%d:%d) = %d:%d %v, but the segment says %d:%d", name, c.out.Path, c.line, c.col, sl, sc, ok, c.srcLine, c.srcCol)
                continue
            }
            o, line, col, ok := res.Locate(sl, sc)
            same := ok && o == c.out && line == c.line && col == c.col
            switch {
            case !ok:
                t.Errorf("%s: Locate(Map(%s:%d:%d)) finds nothing", name, c.out.Path, c.line, c.col)
            case c.first && !same:
                t.Errorf("%s: Locate(Map(%s:%d:%d)) = %s:%d:%d", name, c.out.Path, c.line, c.col, o.Path, line, col)
            case !c.first:
                copies++
                back, backCol, _, _ := o.Map(line, col)
                earlier := o == c.out && (line < c.line || line == c.line && col < c.col)
                if same || !earlier || back != sl || backCol != sc {
                    t.Errorf("%s: from the later copy %d:%d, Locate(Map) = %d:%d, which is not an earlier copy of %d:%d", name, c.line, c.col, line, col, sl, sc)
                }
            }
        }
        if (copies > 0) != (name == "twice") {
            t.Errorf("%s: %d characters are later copies", name, copies)
        }
    }
}

Last come the positions outside the invariant, one test for each row of the table. They’re here so that changing any of them is a decision somebody makes, not an accident.

func TestWhereTheInverseStops(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    res := d.Tangle()
    o := res.Files[0]

    // Generated text: Map answers with the nearest thing, so many positions
    // share one answer and Locate can return to at most one of them.
    var body int
    for n, l := range o.Lines {
        if l.Text == "\t\t\tfmt.Println(i)" {
            body = n
        }
    }
    sl, sc, _, _ := o.Map(body, 2) // the first byte of the chunk's own text
    for col := 0; col < 2; col++ { // the indentation tangling put before it
        if l, c, _, ok := o.Map(body, col); !ok || l != sl || c != sc {
            t.Errorf("Map(%d:%d) = %d:%d, want the nearest segment's start %d:%d", body, col, l, c, sl, sc)
        }
    }
    if _, line, col, _ := res.Locate(sl, sc); line != body || col != 2 {
        t.Errorf("Locate returns to the text at %d:2, not the indentation, got %d:%d", body, line, col)
    }

    // Text that isn't tangled: Locate has no answer, and Map never gives one.
    // A position has an answer if it is a tangled character or the gap
    // after one, and those are all the answers there are. Even a directive
    // is only tangled in part: its value is, the markup around it isn't.
    written := map[[2]int]bool{}
    for _, c := range chars(res) {
        written[[2]int{c.srcLine, c.srcCol}] = true
    }
    prose := 0
    for n, l := range d.Lines {
        for col := 0; col <= len(l); col++ {
            want := written[[2]int{n, col}] || written[[2]int{n, col - 1}]
            if _, _, _, ok := res.Locate(n, col); ok != want {
                t.Errorf("Locate(%d:%d) = %v in %q", n, col, ok, l)
            }
            if !want {
                prose++
            }
        }
    }
    if prose == 0 {
        t.Error("the sample has no prose to fail on")
    }

    // The end of a segment: Locate takes it, Map steps back or aside, and
    // Exact is the one that returns.
    for _, c := range chars(res) {
        if !c.segmentEnd || !c.first {
            continue
        }
        oo, line, col, ok := res.Locate(c.srcLine, c.srcCol+1)
        if !ok || oo != c.out || line != c.line || col != c.col+1 {
            t.Errorf("Locate(%d:%d), the end of a segment, = %d:%d %v", c.srcLine, c.srcCol+1, line, col, ok)
            continue
        }
        if l, cc, _, _ := oo.Map(line, col); l == c.srcLine && cc == c.srcCol+1 {
            t.Errorf("Map(%d:%d) returned to the end of a segment; the table under \"Two halves of one map\" is out of date", line, col)
        }
        if l, cc, ok := oo.Exact(line, col, true); !ok || l != c.srcLine || cc != c.srcCol+1 {
            t.Errorf("Exact(%d:%d, end) = %d:%d %v, want %d:%d", line, col, l, cc, ok, c.srcLine, c.srcCol+1)
        }
    }
}

Mentions, problems and edits

func TestMentionsAndImports(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    name := func(m *Mention) string {
        if m == nil {
            return ""
        }
        return d.Lines[m.Line][m.Col:m.EndCol]
    }
    if m := d.MentionAt(11, 0); name(m) != "body" || !m.Ref {
        t.Errorf("a reference owns its line: %+v", m)
    }
    if m := d.MentionAt(13, 20); name(m) != "args" || !m.Ref {
        t.Errorf("an inline reference: %+v", m)
    }
    if m := d.MentionAt(13, 3); m != nil {
        t.Errorf("fmt is Go, not a chunk: %+v", m)
    }
    if m := d.MentionAt(17, 0); name(m) != "body" || m.Ref || len(m.Chunk.Refs) != 1 {
        t.Errorf("a chunk directive: %+v", m)
    }
    if m := d.MentionAt(21, 3); m != nil {
        t.Errorf("the inside of a chunk is Go too: %+v", m)
    }
    line, col, text, ok := d.AddImport(d.Files[0], "", "strings")
    if !ok || d.Lines[line][:col]+text != "<!-- imports: fmt, os, strings" {
        t.Errorf("AddImport = %d:%d %q", line, col, text)
    }
}

func TestProblems(t *testing.T) {
    cases := map[string]string{
        "```go\n// <<nope>>\n```\n":                                       "not defined",
        "```go\n// <<a>>\n```\n<!-- chunk: a -->\n```go\n// <<a>>\n```\n": "includes itself",
        "```go\nx\n```\n<!-- chunk: lonely -->\n```go\ny\n```\n":          "never used",
        "<!-- chunk: dangling -->\n\nprose\n":                             "not followed by a code block",
        "<!-- package: main -->\n```go\nfunc main() {\n\tx := \n}\n```\n": "",
    }
    for src, want := range cases {
        r := Parse("p.lit.md", []byte(src)).Tangle()
        diags := append(r.Diags, r.SyntaxCheck()...)
        found := false
        for _, dg := range diags {
            if want != "" && strings.Contains(dg.Message, want) {
                found = true
            }
            if want == "" && dg.Source == "syntax" && dg.Line == 4 && len(diags) == 1 {
                found = true
            }
        }
        if !found {
            t.Errorf("%q: wanted %q in %+v", src, want, diags)
        }
    }
}

func TestUntangleRoundTrip(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    before := program(d)
    find := func(prefix string) int {
        for i, l := range d.Lines {
            if strings.HasPrefix(strings.TrimSpace(l), prefix) {
                return i
            }
        }
        t.Fatalf("no line %q", prefix)
        return -1
    }

    // Linewise: the whole for loop.
    start := find("for i :=")
    res, err := d.Untangle(Selection{StartLine: start, EndLine: start + 2, EndCol: -1}, "")
    if err != nil {
        t.Fatal(err)
    }
    if res.Name != "loop i to 3" {
        t.Errorf("name = %q", res.Name)
    }
    lines := Apply(d.Lines, res.Edits)
    d2 := Parse("x.lit.md", []byte(strings.Join(lines, "\n")))
    if after := program(d2); after != before {
        t.Errorf("linewise untangle changed the program:\n%s\n---\n%s", before, after)
    }
    if lines[res.RefLine] != "\t// <<loop i to 3>>" || !strings.HasPrefix(lines[res.DefLine], "for i :=") {
        t.Errorf("ref %q def %q", lines[res.RefLine], lines[res.DefLine])
    }

    // Charwise: the call inside Println.
    ln := find("fmt.Println(add(")
    col := strings.Index(d.Lines[ln], "add(")
    res, err = d.Untangle(Selection{StartLine: ln, EndLine: ln, StartCol: col, EndCol: len(d.Lines[ln]) - 1}, "")
    if err != nil {
        t.Fatal(err)
    }
    if !res.Inline || res.Name != "add call" {
        t.Errorf("inline=%v name=%q", res.Inline, res.Name)
    }
    d3 := Parse("x.lit.md", []byte(strings.Join(Apply(d.Lines, res.Edits), "\n")))
    if after := program(d3); after != before {
        t.Errorf("charwise untangle changed the program:\n%s", after)
    }

    if _, err := d.Untangle(Selection{StartLine: 5, EndLine: 5, EndCol: -1}, ""); err == nil {
        t.Error("untangling prose must fail")
    }
}

func TestRename(t *testing.T) {
    d := Parse("x.lit.md", []byte(sample))
    before := program(d)
    edits, err := d.Rename("args", "the two operands")
    if err != nil {
        t.Fatal(err)
    }
    text := strings.Join(Apply(d.Lines, edits), "\n")
    if !strings.Contains(text, "add(/*<<the two operands>>*/)") || !strings.Contains(text, "<!-- chunk: the two operands -->") {
        t.Errorf("rename missed something:\n%s", text)
    }
    if after := program(Parse("x.lit.md", []byte(text))); after != before {
        t.Error("rename changed the program")
    }
    if _, err := d.Rename("args", "body"); err == nil {
        t.Error("renaming onto an existing chunk must fail")
    }
    if _, err := d.Rename("ghost", "x"); err == nil {
        t.Error("renaming a missing chunk must fail")
    }
}

func TestSuggestName(t *testing.T) {
    cases := []struct {
        code   string
        inline bool
        want   string
    }{
        {"// Feed jobs into the channel.\nfor i := 0; i < m; i++ {", false, "feed jobs into the channel"},
        {"for job := range jobs {", false, "for each job in jobs"},
        {"for _, v := range xs {", false, "for each v in xs"},
        {"for {", false, "loop forever"},
        {"if err != nil {\n\treturn err\n}", false, "handle err"},
        {"if x, ok := m[k]; ok && x > 3 {", false, "if ok && x > 3"},
        {"select {\ncase <-ctx.Done():\n\treturn\ncase job := <-jobs:", false, "select on ctx.Done(), jobs"},
        {"switch v := x.(type) {", false, "type switch on x"},
        {"defer wg.Done()", false, "defer wg.Done"},
        {"go func() {", false, "spawn goroutine"},
        {"go worker(ctx, i)", false, "spawn worker"},
        {"jobs := make(chan int, m)", false, "make jobs"},
        {"ctx, cancel := context.WithCancel(parent)", false, "ctx, cancel from context.WithCancel"},
        {"data, err := os.ReadFile(p)", false, "data from os.ReadFile"},
        {"_, err = w.Write(b)", false, "call w.Write"},
        {"wg.Wait()", false, "call wg.Wait"},
        {"results <- job * 2", false, "send to results"},
        {"func (s *Server) Serve(l net.Listener) error {", false, "func Serve"},
        {"type Job struct {", false, "type Job"},
        {"return nil, fmt.Errorf(\"x\")", false, "return nil, fmt.Errorf(\"x\")"},
        {"context.WithTimeout(ctx, time.Second)", true, "context.WithTimeout call"},
        {"&http.Server{Addr: addr}", true, "http.Server literal"},
        {"func(a, b int) bool { return a < b }", true, "func literal"},
        {"a + b*c", true, "a b c expression"},
    }
    for _, c := range cases {
        if got := SuggestName(strings.Split(c.code, "\n"), c.inline); got != c.want {
            t.Errorf("SuggestName(%q) = %q, want %q", c.code, got, c.want)
        }
    }
}

func TestAnalyzeExample(t *testing.T) {
    src, err := os.ReadFile("../../examples/worker_pool.lit.md")
    if err != nil {
        t.Skip(err)
    }
    a := Parse("worker_pool.lit.md", src).Analyze(AnalyzeOptions{})
    kinds := map[string]int{}
    for _, it := range a.Items {
        kinds[it.Kind]++
    }
    if kinds["mermaid"] != 2 || kinds["math-block"] != 1 || kinds["math"] < 6 || kinds["chunk-def"] != 7 || kinds["chunk-ref"] != 7 {
        t.Errorf("items: %v", kinds)
    }
    if len(a.Diagnostics) != 0 {
        t.Errorf("diagnostics: %+v", a.Diagnostics)
    }
}

func TestProofMath(t *testing.T) {
    for formula, want := range map[string]string{
        "lo*lo <= n ^ n < hi*hi":                "𝑙𝑜² ≤ 𝑛 ∧ 𝑛 < ℎ𝑖²",
        "x * x*x >= 0 ^ y*x*x = a/x*x":          "𝑥³ ≥ 0 ∧ 𝑦 · 𝑥² = 𝑎/𝑥 · 𝑥",
        "n < (r+1)*(r+1) ^ (i+1)*i = f(x)*(x)":  "𝑛 < (𝑟 + 1)² ∧ (𝑖 + 1) · 𝑖 = 𝑓(𝑥) · (𝑥)",
        "Forall k in [0, n) : (k)*(k) >= 0":     "∀𝑘 ∈ [0, 𝑛) : (𝑘)² ≥ 0",
        "f*f(x) + A*A[i] + x*xs + x'*x'":        "𝑓 · 𝑓(𝑥) + 𝐴 · 𝐴[𝑖] + 𝑥 · 𝑥𝑠 + 𝑥′ · 𝑥′",
        "at = -1 v 0 <= at < |A|":               "𝑎𝑡 = −1 ∨ 0 ≤ 𝑎𝑡 < |𝐴|",
        "v >= 0 v v' <> v":                      "𝑣 ≥ 0 ∨ 𝑣′ ≠ 𝑣",
        "~found -> x <> A[:]":                   "¬𝑓𝑜𝑢𝑛𝑑 → 𝑥 ≠ 𝐴[:]",
        "Forall k in [0, |A|) : A[k] <> x":      "∀𝑘 ∈ [0, |𝐴|) : 𝐴[𝑘] ≠ 𝑥",
        "Exists k in Z : a <-> b && c || !d":    "∃𝑘 ∈ ℤ : 𝑎 ↔ 𝑏 ∧ 𝑐 ∨ ¬𝑑",
        "Sorted(A) ::= A[0:n) <= m":             "𝑆𝑜𝑟𝑡𝑒𝑑(𝐴) ≔ 𝐴[0:𝑛) ≤ 𝑚",
        "q * d + A[i] % d = A[i] ^ ret_0 == 1":  "𝑞 · 𝑑 + 𝐴[𝑖] mod 𝑑 = 𝐴[𝑖] ∧ 𝑟𝑒𝑡_0 = 1",
        "s.OutCol <= c ^ c <= s.OutCol + s.Len": "𝑠.𝑂𝑢𝑡𝐶𝑜𝑙 ≤ 𝑐 ∧ 𝑐 ≤ 𝑠.𝑂𝑢𝑡𝐶𝑜𝑙 + 𝑠.𝐿𝑒𝑛",
        "Forall k in [0, ...) : k >= 0":         "∀𝑘 ∈ [0, ∞) : 𝑘 ≥ 0",
        "":                                      "",
    } {
        if got := proofMath(formula, AnalyzeOptions{}); got != want {
            t.Errorf("proofMath(%q) = %q, want %q", formula, got, want)
        }
    }
    // The item covers the formula and nothing else, so that the keyword stays.
    d := Parse("t.lit.md", []byte("```go\n\t//@ Invariant lo <= hi  \n\t//@ a ^ b\n\t// @ Ensures b\n\t//@   c\n```\n"))
    var got []string
    for _, it := range d.Analyze(AnalyzeOptions{}).Items {
        if it.Kind == "vego-math" {
            got = append(got, fmt.Sprintf("%d:%d-%d %s", it.Line, it.Col, it.EndCol, it.Text))
        }
    }
    want := []string{"1:15-23 𝑙𝑜 ≤ ℎ𝑖", "2:5-10 𝑎 ∧ 𝑏", "3:14-15 𝑏", "4:7-8 𝑐"}
    if strings.Join(got, "|") != strings.Join(want, "|") {
        t.Errorf("items = %q, want %q", got, want)
    }
}

func TestTables(t *testing.T) {
    src := "| Key | Does | Cost |\n|---|:-:|--:|\n| `\\r` | **run** it, see [docs](x) | $x^2$ |\n| a \\| b | short |\n\nnot | a table\n\n```go\n// | a | b |\n// |---|---|\n```\n"
    d := Parse("t.lit.md", []byte(src))
    var tables []Item
    for _, it := range d.Analyze(AnalyzeOptions{}).Items {
        if it.Kind == "table" {
            tables = append(tables, it)
        }
    }
    if len(tables) != 1 || tables[0].Line != 0 || tables[0].EndLine != 3 {
        t.Fatalf("tables = %+v", tables)
    }
    table := tables[0].Table
    var got []string
    for _, r := range table.Rows {
        for _, c := range r.Cells {
            got = append(got, c.Text)
        }
        got = append(got, fmt.Sprint(len(r.Hide)))
    }
    if want := `Key|Does|Cost|0|\r|run it, see docs|𝑥²|6|a | b|short|1`; strings.Join(got, "|") != want {
        t.Errorf("cells = %s, want %s", strings.Join(got, "|"), want)
    }
    if fmt.Sprint(table.Align) != "[left center right]" || len(table.Rule) != 3 {
        t.Errorf("rule = %v %v", table.Align, table.Rule)
    }
    // "| `\r` | **run** it..." : the first cell is between the pipes at 0 and 7.
    if c := table.Rows[1].Cells[0]; c != (Cell{From: 1, To: 7, Col: 2, EndCol: 6, Text: `\r`}) {
        t.Errorf("cell = %+v", c)
    }
    if h := table.Rows[1].Hide; h[0] != [2]int{2, 3} || h[1] != [2]int{5, 6} {
        t.Errorf("the backticks are hidden: %v", h)
    }
}

func TestInlineMathRules(t *testing.T) {
    d := Parse("m.lit.md", []byte("costs $5 and $10, but $x^2$ and `$y$` and \\$z\\$ and \\(a\\)\n"))
    a := d.Analyze(AnalyzeOptions{})
    var got []string
    for _, it := range a.Items {
        got = append(got, it.Text)
    }
    if strings.Join(got, "|") != "𝑥²|𝑎" {
        t.Errorf("math items = %q", got)
    }
}

const multi = "<!-- file: go.mod -->\n```gomod\nmodule example\n```\n\n" +
    "<!-- file: a/a.go -->\n<!-- package: a -->\n<!-- imports: strings, example.com/z, fmt -->\n```go\nfunc A() { fmt.Println(\"a\") }\n```\n\n" +
    "```lua\n-- ignored: not the language of a.go\n```\n\n" +
    "<!-- file: p/init.lua -->\n```lua\nlocal M = {}\n-- <<body>>\nreturn M\n```\n\n" +
    "<!-- chunk: body -->\n```lua\nM.x = f(--[[<<arg>>]])\n```\n\n<!-- chunk: arg -->\n```lua\n42\n```\n\n" +
    "<!-- file: a/a_test.go -->\n<!-- package: a -->\n<!-- verbatim -->\n```go\nconst s = \"f(/*<<not a ref>>*/)\"\n// <<nor this>>\n```\n"

func TestFilesLanguagesVerbatim(t *testing.T) {
    d := Parse("/m/doc.lit.md", []byte(multi))
    res := d.Tangle()
    if len(res.Diags) != 0 {
        t.Fatalf("diags: %+v", res.Diags)
    }
    got := map[string]string{}
    for _, o := range res.Files {
        got[o.Path] = string(o.Bytes())
    }
    want := map[string]string{
        "/m/go.mod":      "// Code generated by litgo from doc.lit.md; DO NOT EDIT.\n\nmodule example\n",
        "/m/a/a.go":      "// Code generated by litgo from doc.lit.md; DO NOT EDIT.\n\npackage a\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"example.com/z\"\n)\n\nfunc A() { fmt.Println(\"a\") }\n",
        "/m/p/init.lua":  "-- Code generated by litgo from doc.lit.md; DO NOT EDIT.\n\nlocal M = {}\nM.x = f(42)\nreturn M\n",
        "/m/a/a_test.go": "// Code generated by litgo from doc.lit.md; DO NOT EDIT.\n\npackage a\n\nconst s = \"f(/*<<not a ref>>*/)\"\n// <<nor this>>\n",
    }
    if len(got) != len(want) {
        t.Errorf("files: %v", got)
    }
    for p, w := range want {
        if got[p] != w {
            t.Errorf("%s:\n%q\nwant\n%q", p, got[p], w)
        }
    }
    for _, o := range res.Files {
        if !Generated(o.Bytes()) {
            t.Errorf("%s lacks the generated marker", o.Path)
        }
    }

    // Untangling Lua leaves a Lua comment and a lua chunk.
    ln := 0
    for i, l := range d.Lines {
        if l == "local M = {}" {
            ln = i
        }
    }
    u, err := d.Untangle(Selection{StartLine: ln, EndLine: ln, EndCol: -1}, "")
    if err != nil {
        t.Fatal(err)
    }
    lines := Apply(d.Lines, u.Edits)
    if lines[ln] != "-- <<define M>>" || lines[len(lines)-3] != "```lua" {
        t.Errorf("ref %q, fence %q", lines[ln], lines[len(lines)-3])
    }
    edits, err := d.Rename("arg", "the answer")
    if err != nil {
        t.Fatal(err)
    }
    if text := strings.Join(Apply(d.Lines, edits), "\n"); !strings.Contains(text, "f(--[[<<the answer>>]])") {
        t.Errorf("rename lost the Lua comment syntax:\n%s", text)
    }
}

Math in a character grid

A terminal can’t typeset math. Unicode gets you surprisingly far anyway. \sum_{i=1}^{n} i^2 has a perfectly readable one-line form, and display math can be laid out in two dimensions, with box-drawing characters for the fraction bars and brackets:

x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}

Package latex is a small renderer that turns TeX into text. A recursive-descent parser (one function per grammar rule, each calling the next) builds a tree, and a layout pass turns the tree into boxes of text.

The tree

internal/latex/parse.go
type kind int

const (
    kChar      kind = iota // literal character(s) from the source
    kSym                   // a Unicode symbol produced by a command
    kGroup                 // {...}
    kScript                // kids[0] is the base; sub/sup hold the scripts
    kFrac                  // kids: numerator, denominator
    kBinom                 // kids: top, bottom
    kSqrt                  // kids: radicand[, index]
    kFont                  // text: font name; kids: content
    kText                  // text: upright literal text
    kOp                    // text: upright operator name (sin, log)
    kLimOp                 // text: operator that takes limits (lim, max)
    kAccent                // text: combining mark; kids: content
    kMatrix                // env + rows
    kLeftRight             // left, right delimiters around kids
    kSpace                 // text: literal spaces
    kBoxed
)

type node struct {
    kind        kind
    text        string
    kids        []*node
    sub, sup    *node
    env         string
    rows        [][]*node
    left, right string
}
type parser struct {
    src []rune
    pos int
}

func (p *parser) eof() bool { return p.pos >= len(p.src) }

func (p *parser) peek() rune {
    if p.eof() {
        return 0
    }
    return p.src[p.pos]
}

func (p *parser) skipSpace() {
    for !p.eof() && unicode.IsSpace(p.peek()) {
        p.pos++
    }
}

// command reads the name after a backslash.
func (p *parser) command() string {
    p.pos++ // backslash
    start := p.pos
    for !p.eof() && unicode.IsLetter(p.peek()) {
        p.pos++
    }
    if p.pos == start && !p.eof() {
        p.pos++
    }
    name := string(p.src[start:p.pos])
    if !p.eof() && p.peek() == '*' && len(name) > 1 {
        p.pos++
    }
    return name
}

// peekCommand returns the next command name without consuming it.
func (p *parser) peekCommand() string {
    if p.peek() != '\\' {
        return ""
    }
    save := p.pos
    name := p.command()
    p.pos = save
    return name
}

// rawGroup reads {...} verbatim, for \text and environment names.
func (p *parser) rawGroup() string {
    p.skipSpace()
    if p.peek() != '{' {
        return ""
    }
    p.pos++
    depth, start := 1, p.pos
    for !p.eof() {
        switch p.peek() {
        case '\\':
            p.pos++
        case '{':
            depth++
        case '}':
            depth--
            if depth == 0 {
                s := string(p.src[start:p.pos])
                p.pos++
                return s
            }
        }
        p.pos++
    }
    return string(p.src[start:min(p.pos, len(p.src))])
}

Rows and cells are the outermost structure, because & and \\ can show up bare at the top level of display math, not just inside an environment. A sequence stops at anything its caller has to handle: a closing brace, a cell or row separator, \end, or \right.

// rows parses cells separated by & and rows separated by \\ until a
// terminator: '}' , \end, \right or end of input.
func (p *parser) parseRows() [][]*node {
    rows := [][]*node{{}}
    for {
        cell := &node{kind: kGroup, kids: p.parseSeq()}
        last := len(rows) - 1
        rows[last] = append(rows[last], cell)
        switch {
        case p.peek() == '&':
            p.pos++
        case p.peekCommand() == "\\":
            p.command()
            if p.skipSpace(); p.peek() == '[' { // \\[2pt]
                for !p.eof() && p.peek() != ']' {
                    p.pos++
                }
                p.pos++
            }
            rows = append(rows, nil)
        default:
            // Drop a trailing empty row left by a final \\.
            if n := len(rows) - 1; n > 0 && len(rows[n]) == 1 && len(rows[n][0].kids) == 0 {
                rows = rows[:n]
            }
            return rows
        }
    }
}

// parseSeq parses atoms until something a caller must handle.
func (p *parser) parseSeq() []*node {
    var out []*node
    for !p.eof() {
        c := p.peek()
        if c == '}' || c == '&' {
            break
        }
        if c == '\\' {
            if n := p.peekCommand(); n == "\\" || n == "end" || n == "right" {
                break
            }
        }
        if c == '^' || c == '_' {
            p.pos++
            arg := p.parseArg()
            var base *node
            if n := len(out); n > 0 && out[n-1].kind != kSpace {
                base = out[n-1]
                out = out[:n-1]
            } else {
                base = &node{kind: kGroup}
            }
            if base.kind != kScript {
                base = &node{kind: kScript, kids: []*node{base}}
            }
            if c == '^' {
                if base.sup != nil { // x'^2 and friends: concatenate
                    arg = &node{kind: kGroup, kids: []*node{base.sup, arg}}
                }
                base.sup = arg
            } else {
                base.sub = arg
            }
            out = append(out, base)
            continue
        }
        if c == '\'' {
            p.pos++
            out = append(out, &node{kind: kSym, text: "′"})
            continue
        }
        if a := p.parseAtom(); a != nil {
            out = append(out, a)
        }
    }
    return out
}

func (p *parser) parseArg() *node {
    p.skipSpace()
    if p.eof() {
        return &node{kind: kGroup}
    }
    if a := p.parseAtom(); a != nil {
        return a
    }
    return &node{kind: kGroup}
}

func (p *parser) delimiter() string {
    p.skipSpace()
    if p.eof() {
        return ""
    }
    if p.peek() == '\\' {
        name := p.command()
        if s, ok := symbols[name]; ok {
            return s
        }
        return ""
    }
    c := p.peek()
    p.pos++
    if c == '.' {
        return ""
    }
    return string(c)
}

func (p *parser) parseAtom() *node {
    c := p.peek()
    switch {
    case c == '{':
        p.pos++
        kids := p.parseSeq()
        if p.peek() == '}' {
            p.pos++
        }
        return &node{kind: kGroup, kids: kids}
    case c == '\\':
        return p.parseCommand()
    case unicode.IsSpace(c):
        p.skipSpace()
        return nil
    case c == '~':
        p.pos++
        return &node{kind: kSpace, text: " "}
    case c == '}' || c == '&':
        return nil
    }
    p.pos++
    return &node{kind: kChar, text: string(c)}
}

Commands that take arguments get handled one by one. Everything else is looked up in the symbol tables. An unknown command is shown exactly as it was written, because in an editor that beats an error message.

func (p *parser) parseCommand() *node {
    name := p.command()
    switch name {
    case "frac", "dfrac", "tfrac", "cfrac":
        return &node{kind: kFrac, kids: []*node{p.parseArg(), p.parseArg()}}
    case "binom", "dbinom", "tbinom", "choose":
        return &node{kind: kBinom, kids: []*node{p.parseArg(), p.parseArg()}}
    case "sqrt":
        p.skipSpace()
        var index *node
        if p.peek() == '[' {
            p.pos++
            start := p.pos
            for !p.eof() && p.peek() != ']' {
                p.pos++
            }
            index = parse(string(p.src[start:p.pos]))
            p.pos++
        }
        n := &node{kind: kSqrt, kids: []*node{p.parseArg()}}
        if index != nil {
            n.kids = append(n.kids, index)
        }
        return n
    case "text", "textrm", "mbox", "textit", "textbf", "texttt", "textsf", "operatorname":
        font := fontCommands[name]
        if font == "rm" || font == "" {
            return &node{kind: kText, text: p.rawGroup()}
        }
        return &node{kind: kFont, text: font, kids: []*node{{kind: kText, text: p.rawGroup()}}}
    case "left":
        n := &node{kind: kLeftRight, left: p.delimiter()}
        n.kids = []*node{rowsNode(p.parseRows())}
        if p.peekCommand() == "right" {
            p.command()
            n.right = p.delimiter()
        }
        return n
    case "right": // unmatched
        p.delimiter()
        return nil
    case "begin":
        env := strings.TrimSuffix(p.rawGroup(), "*")
        n := &node{kind: kMatrix, env: env}
        if env == "array" || env == "alignedat" || env == "tabular" {
            n.text = p.rawGroup()
        }
        n.rows = p.parseRows()
        if p.peekCommand() == "end" {
            p.command()
            p.rawGroup()
        }
        return n
    case "end":
        p.rawGroup()
        return nil
    case "overset", "stackrel":
        top, base := p.parseArg(), p.parseArg()
        return &node{kind: kScript, kids: []*node{base}, sup: top}
    case "underset":
        bottom, base := p.parseArg(), p.parseArg()
        return &node{kind: kScript, kids: []*node{base}, sub: bottom}
    case "overline", "underline":
        mark := "\u0305"
        if name == "underline" {
            mark = "\u0332"
        }
        return &node{kind: kAccent, text: mark, kids: []*node{p.parseArg()}}
    case "not":
        return &node{kind: kAccent, text: "\u0338", kids: []*node{p.parseArg()}}
    case "boxed", "fbox":
        return &node{kind: kBoxed, kids: []*node{p.parseArg()}}
    case "underbrace", "overbrace", "phantom", "mathstrut", "vphantom", "hphantom", "smash":
        arg := p.parseArg()
        if strings.Contains(name, "phantom") || name == "mathstrut" {
            return nil
        }
        return arg
    case "textcolor", "colorbox":
        p.rawGroup()
        return p.parseArg()
    case "color", "label", "vspace", "hspace":
        p.rawGroup()
        return nil
    case "tag":
        return &node{kind: kText, text: "  (" + p.rawGroup() + ")"}
    case "pmod":
        return &node{kind: kGroup, kids: []*node{{kind: kText, text: " (mod "}, p.parseArg(), {kind: kText, text: ")"}}}
    case "displaystyle", "textstyle", "scriptstyle", "limits", "nolimits", "nonumber", "notag",
        "big", "Big", "bigg", "Bigg", "bigl", "bigr", "Bigl", "Bigr", "biggl", "biggr", "Biggl",
        "Biggr", "bigm", "Bigm", "middle", "hline", "centering", "!", "allowbreak":
        return nil
    case ",", ":", ";", " ", "enspace", "thinspace":
        return &node{kind: kSpace, text: " "}
    case "quad":
        return &node{kind: kSpace, text: "  "}
    case "qquad":
        return &node{kind: kSpace, text: "    "}
    case "\\":
        return nil
    }
    if font, ok := fontCommands[name]; ok {
        return &node{kind: kFont, text: font, kids: []*node{p.parseArg()}}
    }
    if mark, ok := accents[name]; ok {
        return &node{kind: kAccent, text: string(mark), kids: []*node{p.parseArg()}}
    }
    if label, ok := limitOps[name]; ok {
        return &node{kind: kLimOp, text: label}
    }
    if functions[name] {
        if name == "bmod" {
            name = "mod"
        }
        return &node{kind: kOp, text: name}
    }
    if s, ok := symbols[name]; ok {
        return &node{kind: kSym, text: s}
    }
    return &node{kind: kText, text: "\\" + name}
}

// rowsNode wraps parsed rows: a plain group for the usual single cell, an
// implicit aligned environment when the source used & or \\ bare.
func rowsNode(rows [][]*node) *node {
    if len(rows) == 1 && len(rows[0]) == 1 {
        return rows[0][0]
    }
    return &node{kind: kMatrix, env: "aligned", rows: rows}
}

func parse(src string) *node {
    p := &parser{src: []rune(src)}
    rows := p.parseRows()
    for !p.eof() {
        // A stray terminator (unmatched }, \end, \right): skip it and carry
        // on in the same cell rather than losing the rest of the formula.
        switch p.peekCommand() {
        case "end":
            p.command()
            p.rawGroup()
        case "right":
            p.command()
            p.delimiter()
        default:
            p.pos++
        }
        more := p.parseRows()
        lastRow := rows[len(rows)-1]
        lastCell := lastRow[len(lastRow)-1]
        lastCell.kids = append(lastCell.kids, more[0][0].kids...)
        rows[len(rows)-1] = append(lastRow, more[0][1:]...)
        rows = append(rows, more[1:]...)
    }
    return rowsNode(rows)
}

Symbols

internal/latex/symbols.go

Here are the symbol tables. Letters in math are italic, and Unicode has italic letters in its Mathematical Alphanumeric Symbols block. The block has holes, though, wherever a letter had already been encoded somewhere else. (The italic h is the Planck constant, ℎ. Physicists got there first.)

// symbols maps LaTeX commands to Unicode.
var symbols = map[string]string{
    // Greek
    "alpha": "α", "beta": "β", "gamma": "γ", "delta": "δ", "epsilon": "ϵ", "varepsilon": "ε",
    "zeta": "ζ", "eta": "η", "theta": "θ", "vartheta": "ϑ", "iota": "ι", "kappa": "κ",
    "lambda": "λ", "mu": "μ", "nu": "ν", "xi": "ξ", "pi": "π", "varpi": "ϖ", "rho": "ρ",
    "varrho": "ϱ", "sigma": "σ", "varsigma": "ς", "tau": "τ", "upsilon": "υ", "phi": "ϕ",
    "varphi": "φ", "chi": "χ", "psi": "ψ", "omega": "ω",
    "Gamma": "Γ", "Delta": "Δ", "Theta": "Θ", "Lambda": "Λ", "Xi": "Ξ", "Pi": "Π",
    "Sigma": "Σ", "Upsilon": "Υ", "Phi": "Φ", "Psi": "Ψ", "Omega": "Ω",
    // Binary operators
    "times": "×", "div": "÷", "cdot": "·", "pm": "±", "mp": "∓", "ast": "∗", "star": "⋆",
    "circ": "∘", "bullet": "∙", "oplus": "⊕", "ominus": "⊖", "otimes": "⊗", "odot": "⊙",
    "cup": "∪", "cap": "∩", "setminus": "∖", "land": "∧", "wedge": "∧", "lor": "∨", "vee": "∨",
    "sqcup": "⊔", "sqcap": "⊓", "uplus": "⊎", "dagger": "†", "ddagger": "‡",
    // Relations
    "le": "≤", "leq": "≤", "ge": "≥", "geq": "≥", "ne": "≠", "neq": "≠", "approx": "≈",
    "equiv": "≡", "sim": "∼", "simeq": "≃", "cong": "≅", "propto": "∝", "ll": "≪", "gg": "≫",
    "in": "∈", "notin": "∉", "ni": "∋", "subset": "⊂", "supset": "⊃", "subseteq": "⊆",
    "supseteq": "⊇", "subsetneq": "⊊", "prec": "≺", "succ": "≻", "preceq": "⪯", "succeq": "⪰",
    "mid": "∣", "nmid": "∤", "parallel": "∥", "perp": "⊥", "vdash": "⊢", "dashv": "⊣",
    "models": "⊨", "coloneqq": "≔", "triangleq": "≜", "doteq": "≐",
    // Arrows
    "to": "→", "rightarrow": "→", "leftarrow": "←", "gets": "←", "leftrightarrow": "↔",
    "Rightarrow": "⇒", "Leftarrow": "⇐", "Leftrightarrow": "⇔", "implies": "⟹", "iff": "⟺",
    "impliedby": "⟸", "mapsto": "↦", "longrightarrow": "⟶", "longleftarrow": "⟵",
    "longmapsto": "⟼", "uparrow": "↑", "downarrow": "↓", "updownarrow": "↕", "nearrow": "↗",
    "searrow": "↘", "hookrightarrow": "↪", "rightharpoonup": "⇀", "rightsquigarrow": "⇝",
    "twoheadrightarrow": "↠",
    // Big operators
    "sum": "∑", "prod": "∏", "coprod": "∐", "int": "∫", "iint": "∬", "iiint": "∭", "oint": "∮",
    "bigcup": "⋃", "bigcap": "⋂", "bigoplus": "⨁", "bigotimes": "⨂", "bigvee": "⋁",
    "bigwedge": "⋀", "bigsqcup": "⨆",
    // Misc
    "infty": "∞", "partial": "∂", "nabla": "∇", "forall": "∀", "exists": "∃", "nexists": "∄",
    "neg": "¬", "lnot": "¬", "emptyset": "∅", "varnothing": "∅", "ldots": "…", "dots": "…",
    "cdots": "⋯", "vdots": "⋮", "ddots": "⋱", "therefore": "∴", "because": "∵", "angle": "∠",
    "triangle": "△", "square": "□", "Box": "□", "blacksquare": "■", "qed": "∎", "top": "⊤",
    "bot": "⊥", "ell": "ℓ", "hbar": "ℏ", "Re": "ℜ", "Im": "ℑ", "aleph": "ℵ", "wp": "℘",
    "prime": "′", "degree": "°", "checkmark": "✓", "langle": "⟨", "rangle": "⟩",
    "lfloor": "⌊", "rfloor": "⌋", "lceil": "⌈", "rceil": "⌉", "lvert": "|", "rvert": "|",
    "lVert": "‖", "rVert": "‖", "Vert": "‖", "vert": "|", "backslash": "\\", "lbrace": "{",
    "rbrace": "}", "lbrack": "[", "rbrack": "]", "colon": ":", "surd": "√", "sharp": "♯",
    "flat": "♭", "natural": "♮", "clubsuit": "♣", "diamondsuit": "♢", "heartsuit": "♡",
    "spadesuit": "♠", "S": "§", "P": "¶", "dag": "†", "copyright": "©", "pounds": "£",
    "%": "%", "$": "$", "#": "#", "&": "&", "_": "_", "{": "{", "}": "}", "|": "‖",
}

// relations and binary operators get a space on both sides, as TeX would.
var spaced = map[string]bool{}

func init() {
    for _, s := range []string{"=", "<", ">", "≤", "≥", "≠", "≈", "≡", "∼", "≃", "≅", "∝", "≪", "≫",
        "∈", "∉", "∋", "⊂", "⊃", "⊆", "⊇", "⊊", "≺", "≻", "⪯", "⪰", "∣", "∤", "∥", "⊥", "⊢", "⊣", "⊨",
        "≔", "≜", "≐", "→", "←", "↔", "⇒", "⇐", "⇔", "⟹", "⟺", "⟸", "↦", "⟶", "⟵", "⟼", "↪", "⇀",
        "⇝", "↠", "×", "÷", "·", "±", "∓", "∘", "⊕", "⊖", "⊗", "⊙", "∪", "∩", "∖", "∧", "∨", "⊔", "⊓"} {
        spaced[s] = true
    }
}

// functions are set upright and followed by a thin space.
var functions = map[string]bool{
    "sin": true, "cos": true, "tan": true, "cot": true, "sec": true, "csc": true,
    "arcsin": true, "arccos": true, "arctan": true, "sinh": true, "cosh": true, "tanh": true,
    "log": true, "ln": true, "lg": true, "exp": true, "det": true, "dim": true, "ker": true,
    "deg": true, "gcd": true, "hom": true, "arg": true, "Pr": true, "mod": true, "bmod": true,
}

// limitOps take their scripts above and below in display mode.
var limitOps = map[string]string{
    "lim": "lim", "max": "max", "min": "min", "sup": "sup", "inf": "inf", "limsup": "lim sup",
    "liminf": "lim inf", "argmax": "arg max", "argmin": "arg min",
}

var bigOps = map[string]bool{
    "∑": true, "∏": true, "∐": true, "⋃": true, "⋂": true, "⨁": true, "⨂": true, "⋁": true, "⋀": true, "⨆": true,
}

var accents = map[string]rune{
    "hat": 0x0302, "widehat": 0x0302, "tilde": 0x0303, "widetilde": 0x0303, "bar": 0x0304,
    "vec": 0x20D7, "dot": 0x0307, "ddot": 0x0308, "check": 0x030C, "breve": 0x0306,
    "acute": 0x0301, "grave": 0x0300, "mathring": 0x030A,
}

var superscripts = map[rune]rune{
    '0': '⁰', '1': '¹', '2': '²', '3': '³', '4': '⁴', '5': '⁵', '6': '⁶', '7': '⁷', '8': '⁸', '9': '⁹',
    '+': '⁺', '-': '⁻', '−': '⁻', '=': '⁼', '(': '⁽', ')': '⁾',
    'a': 'ᵃ', 'b': 'ᵇ', 'c': 'ᶜ', 'd': 'ᵈ', 'e': 'ᵉ', 'f': 'ᶠ', 'g': 'ᵍ', 'h': 'ʰ', 'i': 'ⁱ',
    'j': 'ʲ', 'k': 'ᵏ', 'l': 'ˡ', 'm': 'ᵐ', 'n': 'ⁿ', 'o': 'ᵒ', 'p': 'ᵖ', 'r': 'ʳ', 's': 'ˢ',
    't': 'ᵗ', 'u': 'ᵘ', 'v': 'ᵛ', 'w': 'ʷ', 'x': 'ˣ', 'y': 'ʸ', 'z': 'ᶻ',
    'A': 'ᴬ', 'B': 'ᴮ', 'D': 'ᴰ', 'E': 'ᴱ', 'G': 'ᴳ', 'H': 'ᴴ', 'I': 'ᴵ', 'J': 'ᴶ', 'K': 'ᴷ',
    'L': 'ᴸ', 'M': 'ᴹ', 'N': 'ᴺ', 'O': 'ᴼ', 'P': 'ᴾ', 'R': 'ᴿ', 'T': 'ᵀ', 'U': 'ᵁ', 'V': 'ⱽ', 'W': 'ᵂ',
    'α': 'ᵅ', 'β': 'ᵝ', 'γ': 'ᵞ', 'δ': 'ᵟ', 'ε': 'ᵋ', 'θ': 'ᶿ', 'ι': 'ᶥ', 'φ': 'ᵠ', 'ϕ': 'ᶲ', 'χ': 'ᵡ',
    '′': '′', '∗': '*', '*': '*', '∘': '°', '⊤': 'ᵀ', ' ': ' ',
}

var subscripts = map[rune]rune{
    '0': '₀', '1': '₁', '2': '₂', '3': '₃', '4': '₄', '5': '₅', '6': '₆', '7': '₇', '8': '₈', '9': '₉',
    '+': '₊', '-': '₋', '−': '₋', '=': '₌', '(': '₍', ')': '₎',
    'a': 'ₐ', 'e': 'ₑ', 'h': 'ₕ', 'i': 'ᵢ', 'j': 'ⱼ', 'k': 'ₖ', 'l': 'ₗ', 'm': 'ₘ', 'n': 'ₙ',
    'o': 'ₒ', 'p': 'ₚ', 'r': 'ᵣ', 's': 'ₛ', 't': 'ₜ', 'u': 'ᵤ', 'v': 'ᵥ', 'x': 'ₓ',
    'β': 'ᵦ', 'γ': 'ᵧ', 'ρ': 'ᵨ', 'φ': 'ᵩ', 'ϕ': 'ᵩ', 'χ': 'ᵪ', ' ': ' ',
}

// Alphabet bases in the Mathematical Alphanumeric Symbols block, with the
// letters Unicode had already encoded elsewhere.
type alphabet struct {
    upper, lower, digit rune
    holes               map[rune]rune
}

var alphabets = map[string]alphabet{
    "italic": {0x1D434, 0x1D44E, 0, map[rune]rune{'h': 'ℎ'}},
    "bold":   {0x1D400, 0x1D41A, 0x1D7CE, nil},
    "bb": {0x1D538, 0x1D552, 0x1D7D8, map[rune]rune{
        'C': 'ℂ', 'H': 'ℍ', 'N': 'ℕ', 'P': 'ℙ', 'Q': 'ℚ', 'R': 'ℝ', 'Z': 'ℤ'}},
    "cal": {0x1D49C, 0x1D4B6, 0, map[rune]rune{
        'B': 'ℬ', 'E': 'ℰ', 'F': 'ℱ', 'H': 'ℋ', 'I': 'ℐ', 'L': 'ℒ', 'M': 'ℳ', 'R': 'ℛ',
        'e': 'ℯ', 'g': 'ℊ', 'o': 'ℴ'}},
    "frak": {0x1D504, 0x1D51E, 0, map[rune]rune{
        'C': 'ℭ', 'H': 'ℌ', 'I': 'ℑ', 'R': 'ℜ', 'Z': 'ℨ'}},
    "sf": {0x1D5A0, 0x1D5BA, 0x1D7E2, nil},
    "tt": {0x1D670, 0x1D68A, 0x1D7F6, nil},
}

var fontCommands = map[string]string{
    "mathbb": "bb", "mathbf": "bold", "boldsymbol": "bold", "bm": "bold", "mathcal": "cal",
    "mathscr": "cal", "mathfrak": "frak", "mathsf": "sf", "mathtt": "tt", "mathit": "italic",
    "mathrm": "rm", "text": "rm", "textrm": "rm", "textit": "italic", "textbf": "bold",
    "operatorname": "rm", "mathnormal": "italic", "texttt": "tt", "mbox": "rm",
}

func styled(r rune, font string) rune {
    a, ok := alphabets[font]
    if !ok {
        return r
    }
    if h, ok := a.holes[r]; ok {
        return h
    }
    switch {
    case r >= 'A' && r <= 'Z':
        return a.upper + (r - 'A')
    case r >= 'a' && r <= 'z':
        return a.lower + (r - 'a')
    case r >= '0' && r <= '9' && a.digit != 0:
        return a.digit + (r - '0')
    }
    return r
}

Layout

internal/latex/render.go
// Options control rendering.
type Options struct {
    Display bool // two-dimensional layout; otherwise everything fits one line
    Upright bool // keep Latin letters upright instead of math italic
}

// Render converts LaTeX math to lines of Unicode text.
func Render(src string, o Options) []string {
    b := ctx{display: o.Display, upright: o.Upright}.render(parse(strings.TrimSpace(src)))
    out := make([]string, len(b.lines))
    for i, l := range b.lines {
        out[i] = strings.TrimRight(l, " ")
    }
    if len(out) == 1 {
        out[0] = strings.TrimSpace(squeeze(out[0]))
    }
    return out
}

// Inline is Render for a single line.
func Inline(src string) string { return Render(src, Options{})[0] }

type ctx struct {
    display  bool
    upright  bool
    font     string // "" = default math italic
    inScript bool   // inside a sub/superscript: no operator spacing
}

A box is a few lines of text plus the index of its baseline. Boxes join side by side with their baselines lined up, or stack on top of each other. Widths count characters, not bytes, and combining marks count as zero.

// box is a block of text with a baseline row.
type box struct {
    lines []string
    base  int
}

func text(s string) box { return box{lines: []string{s}} }

func width(s string) int {
    n := 0
    for _, r := range s {
        if !unicode.Is(unicode.Mn, r) && !unicode.Is(unicode.Me, r) {
            n++
        }
    }
    return n
}

func (b box) width() int {
    w := 0
    for _, l := range b.lines {
        w = max(w, width(l))
    }
    return w
}

func (b box) height() int { return len(b.lines) }
func (b box) flat() bool  { return len(b.lines) == 1 }

func padRight(s string, w int) string { return s + strings.Repeat(" ", max(0, w-width(s))) }

func align(s string, w int, mode byte) string {
    gap := max(0, w-width(s))
    switch mode {
    case 'r':
        return strings.Repeat(" ", gap) + s
    case 'c':
        return strings.Repeat(" ", gap/2) + s + strings.Repeat(" ", gap-gap/2)
    }
    return s + strings.Repeat(" ", gap)
}

// hcat joins boxes left to right with their baselines aligned.
func hcat(boxes ...box) box {
    above, below := 0, 0
    for _, b := range boxes {
        above = max(above, b.base)
        below = max(below, b.height()-1-b.base)
    }
    lines := make([]string, above+below+1)
    for _, b := range boxes {
        w := b.width()
        for i := range lines {
            j := i - (above - b.base)
            s := ""
            if j >= 0 && j < b.height() {
                s = b.lines[j]
            }
            lines[i] += padRight(s, w)
        }
    }
    return box{lines: lines, base: above}
}

// vstack piles boxes, aligning each line; base selects the baseline box.
func vstack(mode byte, baseBox int, boxes ...box) box {
    w := 0
    for _, b := range boxes {
        w = max(w, b.width())
    }
    var out box
    for i, b := range boxes {
        if i == baseBox {
            out.base = len(out.lines) + b.base
        }
        for _, l := range b.lines {
            out.lines = append(out.lines, align(l, w, mode))
        }
    }
    return out
}

func squeeze(s string) string {
    for strings.Contains(s, "  ") {
        s = strings.ReplaceAll(s, "  ", " ")
    }
    return s
}
func (c ctx) render(n *node) box {
    if n == nil {
        return text("")
    }
    switch n.kind {
    case kChar:
        return text(c.chars(n.text))
    case kSym, kSpace:
        return text(n.text)
    case kText:
        return text(n.text)
    case kOp, kLimOp:
        return text(n.text)
    case kGroup:
        return c.seq(n.kids)
    case kFont:
        sub := c
        sub.font = n.text
        return sub.seq(n.kids)
    case kAccent:
        return c.accent(n)
    case kFrac:
        return c.frac(n)
    case kBinom:
        return c.binom(n)
    case kSqrt:
        return c.sqrt(n)
    case kScript:
        return c.scripts(n)
    case kLeftRight:
        return c.leftRight(n)
    case kMatrix:
        return c.matrix(n)
    case kBoxed:
        return c.boxed(n)
    }
    return text("")
}

// chars styles literal source characters for the current font.
func (c ctx) chars(s string) string {
    font := c.font
    if font == "" && !c.upright {
        font = "italic"
    }
    var sb strings.Builder
    for _, r := range s {
        switch {
        case r == '-':
            sb.WriteRune('−')
        case font == "rm" || font == "plain":
            sb.WriteRune(r)
        default:
            sb.WriteRune(styled(r, font))
        }
    }
    return sb.String()
}

TeX spaces a formula by the class of each atom. litgo does a small slice of that. Relations and binary operators get a space on both sides, a sign right after an opening bracket gets none, and function names get a space after them.

func isOpen(n *node) bool {
    if n.kind == kChar || n.kind == kSym {
        return strings.ContainsAny(n.text, "([{⟨⌊⌈,;:") || spaced[n.text] || n.text == "+" || n.text == "-"
    }
    return n.kind == kSpace
}

// seq lays out siblings, adding the spacing TeX gives operators.
func (c ctx) seq(kids []*node) box {
    var boxes []box
    for i, k := range kids {
        b := c.render(k)
        if !c.inScript && b.flat() {
            s := b.lines[0]
            core := k
            if k.kind == kScript && len(k.kids) > 0 {
                core = k.kids[0]
            }
            switch {
            case (k.kind == kChar || k.kind == kSym) && (spaced[k.text] || k.text == "=" || k.text == "<" || k.text == ">"):
                s = " " + s + " "
            case k.kind == kChar && (k.text == "+" || k.text == "-"):
                if i > 0 && !isOpen(kids[i-1]) {
                    s = " " + s + " "
                }
            case k.kind == kChar && (k.text == "," || k.text == ";"):
                s += " "
            case core.kind == kOp || core.kind == kLimOp || (core.kind == kSym && (bigOps[core.text] || integrals[core.text])):
                if i > 0 && !isOpen(kids[i-1]) {
                    s = " " + s
                }
                if i+1 < len(kids) && !(kids[i+1].kind == kChar && strings.ContainsAny(kids[i+1].text, "(),")) {
                    s += " "
                }
            }
            b.lines[0] = s
        } else if !c.inScript && (k.kind == kScript && k.kids[0].kind == kSym && bigOps[k.kids[0].text] || k.kind == kScript && k.kids[0].kind == kLimOp) {
            b = hcat(b, text(" "))
        }
        boxes = append(boxes, b)
    }
    if len(boxes) == 0 {
        return text("")
    }
    out := hcat(boxes...)
    if out.flat() {
        out.lines[0] = squeeze(out.lines[0])
    }
    return out
}

func (c ctx) accent(n *node) box {
    b := c.render(n.kids[0])
    if !b.flat() {
        return b
    }
    mark := n.text
    runes := []rune(b.lines[0])
    wide := mark == "\u0305" || mark == "\u0332"
    var sb strings.Builder
    for i, r := range runes {
        sb.WriteRune(r)
        if (wide && r != ' ') || (!wide && i == 0) {
            sb.WriteString(mark)
        }
    }
    return text(sb.String())
}

A fraction is stacked in display mode and flattened to a/b inline, with parentheses added wherever an operand would be misread without them.

// needsParens reports whether a flattened operand would be misread without
// grouping.
func needsParens(s string) bool {
    if width(s) <= 1 {
        return false
    }
    depth := 0
    for _, r := range s {
        switch {
        case r == '(' || r == '[':
            depth++
        case r == ')' || r == ']':
            depth--
        case depth == 0 && (r == ' ' || r == '/' || r == '+' || r == '−' || r == '·'):
            return true
        }
    }
    return false
}

func paren(s string) string {
    if needsParens(s) {
        return "(" + s + ")"
    }
    return s
}

func (c ctx) frac(n *node) box {
    inner := c
    if !c.display || c.inScript {
        inner.display = false
        num, den := inner.render(n.kids[0]), inner.render(n.kids[1])
        return text(paren(strings.TrimSpace(num.lines[0])) + "/" + paren(strings.TrimSpace(den.lines[0])))
    }
    num, den := c.render(n.kids[0]), c.render(n.kids[1])
    w := max(num.width(), den.width()) + 2
    rule := text(strings.Repeat("─", w))
    out := vstack('c', 1, num, rule, den)
    return out
}

func (c ctx) binom(n *node) box {
    if !c.display || c.inScript {
        inner := c
        inner.display = false
        return text("C(" + inner.render(n.kids[0]).lines[0] + ", " + inner.render(n.kids[1]).lines[0] + ")")
    }
    body := vstack('c', 0, c.render(n.kids[0]), c.render(n.kids[1]))
    body.base = 0
    if body.height() == 2 {
        body.base = 1
    }
    return delimit("(", body, ")")
}

func (c ctx) sqrt(n *node) box {
    body := c.render(n.kids[0])
    root := "√"
    if len(n.kids) > 1 {
        plain := c
        plain.font, plain.inScript, plain.display = "plain", true, false
        idx := strings.TrimSpace(plain.render(n.kids[1]).lines[0])
        switch idx {
        case "2":
        case "3":
            root = "∛"
        case "4":
            root = "∜"
        default:
            if s, ok := mapAll(idx, superscripts); ok {
                root = s + "√"
            } else {
                root = idx + "√"
            }
        }
    }
    if body.flat() {
        s := strings.TrimSpace(body.lines[0])
        if width(s) > 1 {
            s = "(" + s + ")"
        }
        return text(root + s)
    }
    w := body.width()
    out := box{base: body.base + 1}
    pad := strings.Repeat(" ", width(root))
    out.lines = append(out.lines, pad+"┌"+strings.Repeat("─", w+2))
    for i, l := range body.lines {
        lead := pad
        if i == body.height()-1 {
            lead = root
        }
        out.lines = append(out.lines, lead+"│ "+padRight(l, w)+" ")
    }
    return out
}

Subscripts and superscripts use the real Unicode script characters when every character in the script has one, and fall back to ^(...) when one is missing. There is a superscript n, for example, but no superscript q. Don’t ask me why. In display mode the big operators get their limits above and below.

func mapAll(s string, table map[rune]rune) (string, bool) {
    var sb strings.Builder
    for _, r := range s {
        m, ok := table[r]
        if !ok {
            return "", false
        }
        sb.WriteRune(m)
    }
    return sb.String(), s != ""
}

// script renders a sub- or superscript on one line: real Unicode scripts when
// every character has one, otherwise an explicit ^(...) or _(...).
func (c ctx) script(n *node, table map[rune]rune, marker string) string {
    if n == nil {
        return ""
    }
    plain := c
    plain.font, plain.inScript, plain.display = "plain", true, false
    if c.font != "" && c.font != "italic" {
        plain.font = c.font // keep \mathbf{x}^{\mathbf{T}} distinctions honest
    }
    if s, ok := mapAll(strings.TrimSpace(plain.render(n).lines[0]), table); ok {
        return s
    }
    rich := c
    rich.inScript, rich.display = true, false
    s := strings.TrimSpace(rich.render(n).lines[0])
    if s == "" {
        return ""
    }
    if width(s) == 1 {
        return marker + s
    }
    return marker + "(" + s + ")"
}

func (c ctx) scripts(n *node) box {
    baseNode := n.kids[0]
    limits := c.display && !c.inScript &&
        (baseNode.kind == kLimOp || (baseNode.kind == kSym && bigOps[baseNode.text]))
    if limits {
        small := c
        small.display, small.inScript = false, true
        parts := []box{}
        baseIdx := 0
        if n.sup != nil {
            parts = append(parts, text(strings.TrimSpace(small.render(n.sup).lines[0])))
            baseIdx = 1
        }
        parts = append(parts, c.render(baseNode))
        if n.sub != nil {
            parts = append(parts, text(strings.TrimSpace(small.render(n.sub).lines[0])))
        }
        return vstack('c', baseIdx, parts...)
    }
    base := c.render(baseNode)
    sup := c.script(n.sup, superscripts, "^")
    sub := c.script(n.sub, subscripts, "_")
    if base.flat() {
        return text(base.lines[0] + sub + sup)
    }
    // Tall base: superscript at the top right, subscript at the bottom right.
    col := box{lines: make([]string, base.height()), base: base.base}
    col.lines[0] = sup
    col.lines[base.height()-1] += sub
    return hcat(base, col)
}

Delimiters grow to fit what they hold. They’re built from the bracket pieces Unicode includes for exactly this job.

var integrals = map[string]bool{"∫": true, "∬": true, "∭": true, "∮": true}

var tall = map[string][4]string{ // top, middle, bottom, centre piece
    "(": {"⎛", "⎜", "⎝", "⎜"}, ")": {"⎞", "⎟", "⎠", "⎟"},
    "[": {"⎡", "⎢", "⎣", "⎢"}, "]": {"⎤", "⎥", "⎦", "⎥"},
    "{": {"⎧", "⎪", "⎩", "⎨"}, "}": {"⎫", "⎪", "⎭", "⎬"},
    "|": {"│", "│", "│", "│"}, "‖": {"║", "║", "║", "║"},
    "⌊": {"│", "│", "⎣", "│"}, "⌋": {"│", "│", "⎦", "│"},
    "⌈": {"⎡", "│", "│", "│"}, "⌉": {"⎤", "│", "│", "│"},
}

func delimiter(d string, h, base int) box {
    if d == "" {
        return text("")
    }
    parts, ok := tall[d]
    if h == 1 || !ok {
        b := box{lines: make([]string, h), base: base}
        for i := range b.lines {
            b.lines[i] = strings.Repeat(" ", width(d))
        }
        b.lines[base] = d
        return b
    }
    if h == 2 && (d == "{" || d == "}") {
        if d == "{" {
            return box{lines: []string{"⎰", "⎱"}, base: base}
        }
        return box{lines: []string{"⎱", "⎰"}, base: base}
    }
    b := box{lines: make([]string, h), base: base}
    for i := range b.lines {
        switch i {
        case 0:
            b.lines[i] = parts[0]
        case h - 1:
            b.lines[i] = parts[2]
        case h / 2:
            b.lines[i] = parts[3]
        default:
            b.lines[i] = parts[1]
        }
    }
    return b
}

func delimit(l string, body box, r string) box {
    return hcat(delimiter(l, body.height(), body.base), body, delimiter(r, body.height(), body.base))
}

func (c ctx) leftRight(n *node) box {
    return delimit(n.left, c.render(n.kids[0]), n.right)
}

func (c ctx) boxed(n *node) box {
    body := c.render(n.kids[0])
    if !c.display {
        return text("[" + body.lines[0] + "]")
    }
    w := body.width()
    out := box{base: body.base + 1}
    out.lines = append(out.lines, "┌"+strings.Repeat("─", w+2)+"┐")
    for _, l := range body.lines {
        out.lines = append(out.lines, "│ "+padRight(l, w)+" │")
    }
    out.lines = append(out.lines, "└"+strings.Repeat("─", w+2)+"┘")
    return out
}

func (c ctx) matrix(n *node) box {
    env := n.env
    delims := map[string][2]string{
        "pmatrix": {"(", ")"}, "bmatrix": {"[", "]"}, "Bmatrix": {"{", "}"},
        "vmatrix": {"|", "|"}, "Vmatrix": {"‖", "‖"}, "cases": {"{", ""},
        "smallmatrix": {"", ""},
    }[env]

    if !c.display || c.inScript {
        var rows []string
        for _, r := range n.rows {
            var cells []string
            for _, cell := range r {
                cells = append(cells, strings.TrimSpace(c.render(cell).lines[0]))
            }
            rows = append(rows, strings.Join(cells, " "))
        }
        return text(delims[0] + strings.Join(rows, "; ") + delims[1])
    }

    ncols := 0
    for _, r := range n.rows {
        ncols = max(ncols, len(r))
    }
    cells := make([][]box, len(n.rows))
    widths := make([]int, ncols)
    for i, r := range n.rows {
        cells[i] = make([]box, ncols)
        for j := range cells[i] {
            cells[i][j] = text("")
            if j < len(r) {
                b := c.render(r[j])
                for k := range b.lines {
                    b.lines[k] = strings.TrimRight(b.lines[k], " ")
                }
                if b.flat() {
                    b.lines[0] = strings.TrimSpace(b.lines[0])
                }
                cells[i][j] = b
            }
            widths[j] = max(widths[j], cells[i][j].width())
        }
    }

    colAlign := func(j int) byte {
        switch env {
        case "cases":
            return 'l'
        case "aligned", "align", "split", "eqnarray", "alignat", "alignedat", "flalign":
            if j%2 == 0 {
                return 'r'
            }
            return 'l'
        case "array":
            spec := strings.Map(func(r rune) rune {
                if r == 'l' || r == 'c' || r == 'r' {
                    return r
                }
                return -1
            }, n.text)
            if j < len(spec) {
                return spec[j]
            }
        case "gathered", "gather", "equation", "multline":
            return 'c'
        }
        return 'c'
    }
    aligned := colAlign(0) == 'r' && env != "array"

    var rowBoxes []box
    for i := range cells {
        var parts []box
        for j, cell := range cells[i] {
            for k := range cell.lines {
                cell.lines[k] = align(cell.lines[k], widths[j], colAlign(j))
            }
            if j > 0 {
                gap := "  "
                if aligned && j%2 == 1 {
                    gap = " "
                }
                parts = append(parts, text(gap))
            }
            parts = append(parts, cell)
        }
        rowBoxes = append(rowBoxes, hcat(parts...))
    }
    body := vstack('l', 0, rowBoxes...)
    body.base = (body.height() - 1) / 2
    return delimit(delims[0], body, delims[1])
}
internal/latex/latex_test.go
func TestInline(t *testing.T) {
    cases := map[string]string{
        `n`:                           "𝑛",
        `O(n \log n)`:                 "𝑂(𝑛 log 𝑛)",
        `x^2 + y_i`:                   "𝑥² + 𝑦ᵢ",
        `\frac{a}{b}`:                 "𝑎/𝑏",
        `\frac{n(n+1)}{2}`:            "𝑛(𝑛 + 1)/2",
        `\sum_{i=1}^{n} i`:            "∑ᵢ₌₁ⁿ 𝑖",
        `\mathbb{R}^n \to \mathbb{N}`: "ℝⁿ → ℕ",
        `a \le b \neq c`:              "𝑎 ≤ 𝑏 ≠ 𝑐",
        `\sqrt{x+1}`:                  "√(𝑥 + 1)",
        `e^{-x}`:                      "𝑒⁻ˣ",
        `x^{q+1}`:                     "𝑥^(𝑞+1)",
        `\alpha\beta`:                 "αβ",
        `f'(x)`:                       "𝑓′(𝑥)",
        `\text{if } x > 0`:            "if 𝑥 > 0",
        `-x`:                          "−𝑥",
        `h`:                           "ℎ",
        `\int_0^1 x\,dx`:              "∫₀¹ 𝑥 𝑑𝑥",
        `\frac{a+b}{c}`:               "(𝑎 + 𝑏)/𝑐",
        `\hat{x} + \bar{y}`:           "𝑥\u0302 + 𝑦\u0304",
    }
    for src, want := range cases {
        if got := Inline(src); got != want {
            t.Errorf("Inline(%q) = %q, want %q", src, got, want)
        }
    }
}

func TestDisplay(t *testing.T) {
    for _, src := range []string{
        `\frac{-b \pm \sqrt{b^2 - 4ac}}{2a}`,
        `\sum_{i=1}^{n} i = \frac{n(n+1)}{2}`,
        `T(n) = \begin{cases} 1 & n = 1 \\ 2T(n/2) + n & \text{otherwise} \end{cases}`,
        `A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}, \quad \left( \frac{x}{y} \right)^2`,
        `\begin{aligned} a &= b + c \\ &= d \end{aligned}`,
        `\lim_{x \to \infty} \frac{1}{x} = 0 \qquad \int_0^1 x\,dx = \frac{1}{2}`,
        `\sqrt{\frac{a}{b}} + \binom{n}{k}`,
    } {
        out := Render(src, Options{Display: true})
        t.Logf("%s\n%s", src, strings.Join(out, "\n"))
        if len(out) == 0 {
            t.Errorf("no output for %q", src)
        }
    }
}

func TestMalformedDoesNotPanic(t *testing.T) {
    for _, src := range []string{``, `}`, `{`, `\frac`, `\frac{a}`, `x^`, `_`, `\left(`, `\right)`, `\begin{cases}`, `\end{x}`, `a & b \\`, `\sqrt[`, `\`} {
        Render(src, Options{Display: true})
        Render(src, Options{})
    }
}

Diagrams in a character grid

The mermaid package draws the three kinds of Mermaid diagram that do the most for explaining a program: flowcharts, state diagrams and sequence diagrams. A flowchart goes through the classic pipeline for layered graph drawing:

flowchart LR
    parse --> rank --> split[split long edges] --> order
    order --> place --> route[assign tracks] --> draw

A canvas of strokes

internal/mermaid/canvas.go

litgo doesn’t draw lines as characters. Each cell records which of its four sides a stroke leaves through, and the character is picked at the very end. So the right junction shows up by itself, wherever an edge meets a box or two edges cross.

// Direction bits of the line strokes leaving a cell.
const (
    up = 1 << iota
    down
    left
    right
)

type lineStyle int

const (
    solid lineStyle = iota
    dotted
    thick
    rounded
    double
)

type cell struct {
    bits  int
    style lineStyle
    r     rune // explicit content; wins over strokes
}

// canvas is a growable character grid. Lines are recorded as strokes so
// crossings and junctions resolve to the right box-drawing character.
type canvas struct {
    cells [][]cell
    w, h  int
}

at hands out pointers into slices that grow might reallocate. So a stroke grows the canvas first, and only then takes pointers.

func (c *canvas) grow(x, y int) {
    for y >= len(c.cells) {
        c.cells = append(c.cells, nil)
    }
    for x >= len(c.cells[y]) {
        c.cells[y] = append(c.cells[y], cell{})
    }
    c.w, c.h = max(c.w, x+1), max(c.h, y+1)
}

func (c *canvas) at(x, y int) *cell {
    if x < 0 || y < 0 {
        return &cell{}
    }
    c.grow(x, y)
    return &c.cells[y][x]
}

func (c *canvas) put(x, y int, r rune) { c.at(x, y).r = r }

func (c *canvas) write(x, y int, s string) {
    for _, r := range s {
        c.put(x, y, r)
        x++
    }
}

// stroke draws a straight line between two cells in the same row or column.
func (c *canvas) stroke(x1, y1, x2, y2 int, st lineStyle) {
    dx, dy := sign(x2-x1), sign(y2-y1)
    if dx != 0 && dy != 0 {
        return
    }
    for x, y := x1, y1; x != x2 || y != y2; x, y = x+dx, y+dy {
        // Grow first: at() hands out pointers that a later grow would orphan.
        c.grow(max(x, x+dx, 0), max(y, y+dy, 0))
        a, b := c.at(x, y), c.at(x+dx, y+dy)
        // The first stroke through a cell decides its style; where styles
        // meet, the glyph lookup falls back to plain junctions.
        for _, p := range []*cell{a, b} {
            if p.bits == 0 {
                p.style = st
            }
        }
        switch {
        case dx > 0:
            a.bits |= right
            b.bits |= left
        case dx < 0:
            a.bits |= left
            b.bits |= right
        case dy > 0:
            a.bits |= down
            b.bits |= up
        default:
            a.bits |= up
            b.bits |= down
        }
    }
}

// rect outlines a box.
func (c *canvas) rect(x, y, w, h int, st lineStyle) {
    c.stroke(x, y, x+w-1, y, st)
    c.stroke(x+w-1, y, x+w-1, y+h-1, st)
    c.stroke(x+w-1, y+h-1, x, y+h-1, st)
    c.stroke(x, y+h-1, x, y, st)
}

func sign(n int) int {
    switch {
    case n > 0:
        return 1
    case n < 0:
        return -1
    }
    return 0
}
var glyphs = map[int]rune{
    up: '│', down: '│', up | down: '│', left: '─', right: '─', left | right: '─',
    down | right: '┌', down | left: '┐', up | right: '└', up | left: '┘',
    up | down | right: '├', up | down | left: '┤', left | right | down: '┬', left | right | up: '┴',
    up | down | left | right: '┼',
}

var styled = map[lineStyle]map[int]rune{
    dotted:  {up | down: '┆', up: '┆', down: '┆', left | right: '┄', left: '┄', right: '┄'},
    thick:   {up | down: '┃', up: '┃', down: '┃', left | right: '━', left: '━', right: '━', down | right: '┏', down | left: '┓', up | right: '┗', up | left: '┛'},
    rounded: {down | right: '╭', down | left: '╮', up | right: '╰', up | left: '╯'},
    double:  {up | down: '║', left | right: '═', down | right: '╔', down | left: '╗', up | right: '╚', up | left: '╝'},
}

func (c *canvas) lines() []string {
    out := make([]string, 0, c.h)
    for _, row := range c.cells {
        var sb strings.Builder
        for _, cl := range row {
            switch {
            case cl.r != 0:
                sb.WriteRune(cl.r)
            case cl.bits != 0:
                if g, ok := styled[cl.style][cl.bits]; ok {
                    sb.WriteRune(g)
                } else {
                    sb.WriteRune(glyphs[cl.bits])
                }
            default:
                sb.WriteByte(' ')
            }
        }
        out = append(out, strings.TrimRight(sb.String(), " "))
    }
    for len(out) > 0 && out[len(out)-1] == "" {
        out = out[:len(out)-1]
    }
    return out
}

Reading a flowchart

internal/mermaid/parse.go
type shape int

const (
    shapeRect shape = iota
    shapeRound
    shapeDiamond
    shapeDouble
    shapeBare // text only, used for state-diagram start and end markers
)

type fnode struct {
    id    string
    label []string
    shape shape
    dummy bool
    loop  bool // has an edge to itself

    rank, order  int
    sizeR, sizeO int // extent along the rank axis and the order axis
    posR, posO   int
}

func (n *fnode) centerO() int { return n.posO + n.sizeO/2 }

type fedge struct {
    from, to *fnode
    label    string
    style    lineStyle
    head     rune // '>' arrow, 'x', 'o', or 0 for an open link
    tailHead bool // <--> also points back at the source
    reversed bool // drawn against the layout direction (a back edge)
}

type graph struct {
    dir   string // TD, BT, LR, RL
    nodes []*fnode
    byID  map[string]*fnode
    edges []*fedge
}

func (g *graph) node(id string) *fnode {
    if n, ok := g.byID[id]; ok {
        return n
    }
    n := &fnode{id: id, label: []string{id}}
    g.byID[id] = n
    g.nodes = append(g.nodes, n)
    return n
}
var (
    headerRe  = regexp.MustCompile(`^(?:graph|flowchart)\s*(TB|TD|BT|LR|RL)?\s*;?\s*$`)
    idRe      = regexp.MustCompile(`^[\p{L}\p{N}_]+`)
    ignoredRe = regexp.MustCompile(`^(style|classDef|class|click|linkStyle|accTitle|accDescr|direction|end\b|subgraph\b)`)
    brRe      = regexp.MustCompile(`(?i)<br\s*/?>|\\n`)
    tagRe     = regexp.MustCompile(`</?[a-zA-Z][^>]*>`)

    linkTextRes = []*regexp.Regexp{
        regexp.MustCompile(`^(<)?--\s*([^\s\->|][^|>]*?)\s*-{2,}([>xo]?)`),
        regexp.MustCompile(`^(<)?-\.\s*([^\s.\-][^|>]*?)\s*\.+-([>xo]?)`),
        regexp.MustCompile(`^(<)?==\s*([^\s=>|][^|>]*?)\s*={2,}([>xo]?)`),
    }
    linkRe     = regexp.MustCompile(`^(<)?(-{2,}|={2,}|-\.+-)([>xo]?)`)
    pipeTextRe = regexp.MustCompile(`^\s*\|([^|]*)\|`)
)

var shapes = []struct {
    open, close string
    shape       shape
}{
    {"(((", ")))", shapeRound}, {"((", "))", shapeRound}, {"([", "])", shapeRound},
    {"[[", "]]", shapeDouble}, {"[(", ")]", shapeRound}, {"[/", "/]", shapeRect},
    {"[/", "\\]", shapeRect}, {"[\\", "\\]", shapeRect}, {"[\\", "/]", shapeRect},
    {"{{", "}}", shapeDiamond}, {"(", ")", shapeRound}, {"[", "]", shapeRect},
    {"{", "}", shapeDiamond}, {">", "]", shapeRect},
}

func cleanLabel(s string) []string {
    s = strings.TrimSpace(s)
    if len(s) >= 2 && s[0] == '"' && s[len(s)-1] == '"' {
        s = s[1 : len(s)-1]
    }
    s = strings.Trim(s, "`")
    s = brRe.ReplaceAllString(s, "\n")
    s = tagRe.ReplaceAllString(s, "")
    s = strings.NewReplacer("#quot;", `"`, "&quot;", `"`, "&amp;", "&", "&lt;", "<", "&gt;", ">", "**", "").Replace(s)
    var out []string
    for _, l := range strings.Split(s, "\n") {
        out = append(out, strings.TrimSpace(l))
    }
    return out
}

A statement is a chain: a group of nodes, a link, another group, another link, and so on. A & B --> C fans in. A --> B --> C keeps the chain going.

type stmt struct {
    g   *graph
    s   string
    pos int
}

func (p *stmt) rest() string { return p.s[p.pos:] }

func (p *stmt) skip() {
    for p.pos < len(p.s) && (p.s[p.pos] == ' ' || p.s[p.pos] == '\t') {
        p.pos++
    }
}

func (p *stmt) node() *fnode {
    p.skip()
    id := idRe.FindString(p.rest())
    if id == "" {
        return nil
    }
    p.pos += len(id)
    n := p.g.node(id)
    rest := p.rest()
    for _, sh := range shapes {
        if !strings.HasPrefix(rest, sh.open) {
            continue
        }
        // Find the closer, skipping over quoted text.
        inQuote := false
        for i := len(sh.open); i < len(rest); i++ {
            if rest[i] == '"' {
                inQuote = !inQuote
            }
            if !inQuote && strings.HasPrefix(rest[i:], sh.close) {
                n.label = cleanLabel(rest[len(sh.open):i])
                n.shape = sh.shape
                p.pos += i + len(sh.close)
                goto shaped
            }
        }
    }
shaped:
    if strings.HasPrefix(p.rest(), ":::") {
        p.pos += 3
        p.pos += len(idRe.FindString(p.rest()))
    }
    return n
}

func (p *stmt) group() []*fnode {
    var out []*fnode
    for {
        n := p.node()
        if n == nil {
            return out
        }
        out = append(out, n)
        p.skip()
        if !strings.HasPrefix(p.rest(), "&") {
            return out
        }
        p.pos++
    }
}

func (p *stmt) link() *fedge {
    p.skip()
    rest := p.rest()
    styleOf := func(s string) lineStyle {
        switch {
        case strings.HasPrefix(s, "="):
            return thick
        case strings.Contains(s, "."):
            return dotted
        }
        return solid
    }
    head := func(s string) rune {
        if s == "" {
            return 0
        }
        return rune(s[0])
    }
    for i, re := range linkTextRes {
        if m := re.FindStringSubmatch(rest); m != nil {
            p.pos += len(m[0])
            return &fedge{label: strings.Join(cleanLabel(m[2]), " "), style: []lineStyle{solid, dotted, thick}[i], head: head(m[3]), tailHead: m[1] != ""}
        }
    }
    if m := linkRe.FindStringSubmatch(rest); m != nil {
        p.pos += len(m[0])
        e := &fedge{style: styleOf(m[2]), head: head(m[3]), tailHead: m[1] != ""}
        if t := pipeTextRe.FindStringSubmatch(p.rest()); t != nil {
            p.pos += len(t[0])
            e.label = strings.Join(cleanLabel(t[1]), " ")
        }
        return e
    }
    return nil
}

func (g *graph) statement(s string) {
    p := &stmt{g: g, s: s}
    from := p.group()
    for len(from) > 0 {
        e := p.link()
        if e == nil {
            return
        }
        to := p.group()
        for _, a := range from {
            for _, b := range to {
                edge := *e
                edge.from, edge.to = a, b
                g.edges = append(g.edges, &edge)
            }
        }
        from = to
    }
}

func statements(src string) []string {
    var out []string
    for _, line := range strings.Split(src, "\n") {
        if i := strings.Index(line, "%%"); i >= 0 {
            line = line[:i]
        }
        // Semicolons separate statements, except inside quotes or brackets.
        depth, inQuote, start := 0, false, 0
        for i, r := range line {
            switch {
            case r == '"':
                inQuote = !inQuote
            case inQuote:
            case strings.ContainsRune("[({", r):
                depth++
            case strings.ContainsRune("])}", r):
                depth--
            case r == ';' && depth <= 0:
                out = append(out, strings.TrimSpace(line[start:i]))
                start = i + 1
            }
        }
        out = append(out, strings.TrimSpace(line[start:]))
    }
    var kept []string
    for _, s := range out {
        if s != "" {
            kept = append(kept, s)
        }
    }
    return kept
}

func parseFlowchart(lines []string) (*graph, error) {
    g := &graph{dir: "TD", byID: map[string]*fnode{}}
    if m := headerRe.FindStringSubmatch(lines[0]); m != nil && m[1] != "" {
        g.dir = m[1]
    }
    if g.dir == "TB" {
        g.dir = "TD"
    }
    for _, s := range lines[1:] {
        if ignoredRe.MatchString(s) {
            continue
        }
        g.statement(s)
    }
    if len(g.nodes) == 0 {
        return nil, fmt.Errorf("flowchart has no nodes")
    }
    return g, nil
}

A state diagram is a flowchart with different punctuation. That’s it.

var (
    stateEdgeRe  = regexp.MustCompile(`^(\[\*\]|[\p{L}\p{N}_]+)\s*-->\s*(\[\*\]|[\p{L}\p{N}_]+)\s*(?::\s*(.*))?$`)
    stateAliasRe = regexp.MustCompile(`^state\s+"([^"]*)"\s+as\s+([\p{L}\p{N}_]+)`)
    stateDescRe  = regexp.MustCompile(`^([\p{L}\p{N}_]+)\s*:\s*(.+)$`)
    stateDirRe   = regexp.MustCompile(`^direction\s+(TB|TD|BT|LR|RL)`)
)

func parseState(lines []string) (*graph, error) {
    g := &graph{dir: "TD", byID: map[string]*fnode{}}
    inNote := false
    for _, s := range lines[1:] {
        switch {
        case inNote:
            inNote = !strings.HasPrefix(s, "end note")
        case strings.HasPrefix(s, "note "):
            inNote = !strings.Contains(s, ":")
        case stateDirRe.MatchString(s):
            if g.dir = stateDirRe.FindStringSubmatch(s)[1]; g.dir == "TB" {
                g.dir = "TD"
            }
        case stateAliasRe.MatchString(s):
            m := stateAliasRe.FindStringSubmatch(s)
            n := g.node(m[2])
            n.label, n.shape = cleanLabel(m[1]), shapeRound
        case stateEdgeRe.MatchString(s):
            m := stateEdgeRe.FindStringSubmatch(s)
            end := func(id string, start bool) *fnode {
                if id != "[*]" {
                    n := g.node(id)
                    if n.shape == shapeRect {
                        n.shape = shapeRound
                    }
                    return n
                }
                n := g.node("[*]end")
                n.label = []string{"◉"}
                if start {
                    n = g.node("[*]start")
                    n.label = []string{"●"}
                }
                n.shape = shapeBare
                return n
            }
            g.edges = append(g.edges, &fedge{from: end(m[1], true), to: end(m[2], false),
                label: strings.Join(cleanLabel(m[3]), " "), head: '>'})
        case stateDescRe.MatchString(s):
            m := stateDescRe.FindStringSubmatch(s)
            n := g.node(m[1])
            n.label, n.shape = cleanLabel(m[2]), shapeRound
        }
    }
    if len(g.nodes) == 0 {
        return nil, fmt.Errorf("state diagram has no states")
    }
    return g, nil
}

// Render draws Mermaid source as text.
func Render(src string) ([]string, error) {
    lines := statements(src)
    if len(lines) == 0 {
        return nil, fmt.Errorf("empty diagram")
    }
    kind := strings.Fields(lines[0])[0]
    switch {
    case kind == "graph" || kind == "flowchart" || strings.HasPrefix(kind, "flowchart"):
        g, err := parseFlowchart(lines)
        if err != nil {
            return nil, err
        }
        return g.render(), nil
    case strings.HasPrefix(kind, "stateDiagram"):
        g, err := parseState(lines)
        if err != nil {
            return nil, err
        }
        return g.render(), nil
    case kind == "sequenceDiagram":
        return renderSequence(lines[1:])
    }
    return nil, fmt.Errorf("%s diagrams are not rendered in the terminal yet (flowchart, stateDiagram and sequenceDiagram are)", kind)
}

Layout

internal/mermaid/layout.go

The layout works on two abstract axes. The rank axis is the direction the graph flows, and the order axis runs across it. Top-down and left-to-right are the same computation with the axes swapped at drawing time, and bottom-up and right-to-left are mirror images of those two.

// seg is one rank-to-rank hop of an edge; long edges become chains of
// segments through dummy nodes.
type seg struct {
    from, to    *fnode
    edge        *fedge
    first, last bool
    track       int
}

type layout struct {
    g        *graph
    vertical bool // ranks stack top-to-bottom (TD/BT) rather than left-to-right
    mirror   bool // BT and RL: flip the rank axis
    ranks    [][]*fnode
    segs     [][]*seg // by gap: segs[r] joins rank r to rank r+1
    preds    map[*fnode][]*fnode
    succs    map[*fnode][]*fnode

    rankStart, rankSize []int
    gapStub, gapTracks  []int
    gapLabel            []int
    totalR              int
}

func (g *graph) render() []string {
    l := &layout{g: g, vertical: g.dir == "TD" || g.dir == "BT", mirror: g.dir == "BT" || g.dir == "RL",
        preds: map[*fnode][]*fnode{}, succs: map[*fnode][]*fnode{}}
    l.rank()
    l.split()
    l.order()
    l.measure()
    l.placeO()
    l.tracks()
    l.placeR()
    return l.draw()
}

Cycles come first. The layout reverses the back edges while it computes ranks (they still get drawn with the arrowhead at the right end), and then a node’s rank is the length of its longest path from a source.

// rank breaks cycles, then assigns each node its longest-path depth.
func (l *layout) rank() {
    g := l.g
    var edges []*fedge
    for _, e := range g.edges {
        if e.from == e.to {
            e.from.loop = true
            continue
        }
        edges = append(edges, e)
    }
    g.edges = edges

    out := map[*fnode][]*fedge{}
    for _, e := range g.edges {
        out[e.from] = append(out[e.from], e)
    }
    state := map[*fnode]int{}
    var visit func(n *fnode)
    visit = func(n *fnode) {
        state[n] = 1
        for _, e := range out[n] {
            switch state[e.to] {
            case 0:
                visit(e.to)
            case 1: // back edge: lay it out reversed, draw the arrow the right way
                e.reversed = true
            }
        }
        state[n] = 2
    }
    for _, n := range g.nodes {
        if state[n] == 0 {
            visit(n)
        }
    }
    for _, e := range g.edges {
        if e.reversed {
            e.from, e.to = e.to, e.from
        }
    }

    indeg := map[*fnode]int{}
    succ := map[*fnode][]*fnode{}
    for _, e := range g.edges {
        indeg[e.to]++
        succ[e.from] = append(succ[e.from], e.to)
    }
    var queue, topo []*fnode
    for _, n := range g.nodes {
        if indeg[n] == 0 {
            queue = append(queue, n)
        }
    }
    for len(queue) > 0 {
        n := queue[0]
        queue = queue[1:]
        topo = append(topo, n)
        for _, m := range succ[n] {
            m.rank = max(m.rank, n.rank+1)
            if indeg[m]--; indeg[m] == 0 {
                queue = append(queue, m)
            }
        }
    }
    // Pull sources down next to what they feed, so they do not float at the top.
    hasPred := map[*fnode]bool{}
    for _, e := range g.edges {
        hasPred[e.to] = true
    }
    for i := len(topo) - 1; i >= 0; i-- {
        n := topo[i]
        if hasPred[n] || len(succ[n]) == 0 {
            continue
        }
        lowest := succ[n][0].rank
        for _, m := range succ[n] {
            lowest = min(lowest, m.rank)
        }
        n.rank = lowest - 1
    }
}

// split inserts dummy nodes so every segment spans exactly one rank.
func (l *layout) split() {
    maxRank := 0
    for _, n := range l.g.nodes {
        maxRank = max(maxRank, n.rank)
    }
    l.ranks = make([][]*fnode, maxRank+1)
    l.segs = make([][]*seg, maxRank)
    for _, n := range l.g.nodes {
        l.ranks[n.rank] = append(l.ranks[n.rank], n)
    }
    for _, e := range l.g.edges {
        prev := e.from
        for r := e.from.rank + 1; r <= e.to.rank; r++ {
            next := e.to
            if r < e.to.rank {
                next = &fnode{dummy: true, rank: r}
                l.ranks[r] = append(l.ranks[r], next)
            }
            l.segs[r-1] = append(l.segs[r-1], &seg{from: prev, to: next, edge: e, first: prev == e.from, last: next == e.to})
            l.preds[next] = append(l.preds[next], prev)
            l.succs[prev] = append(l.succs[prev], next)
            prev = next
        }
    }
}

Inside a rank, the nodes are ordered to cut down on crossings. The layout keeps sorting each rank by the average position of each node’s neighbours in the rank next to it.

func (l *layout) crossings() int {
    total := 0
    for _, segs := range l.segs {
        for i, a := range segs {
            for _, b := range segs[i+1:] {
                if (a.from.order-b.from.order)*(a.to.order-b.to.order) < 0 {
                    total++
                }
            }
        }
    }
    return total
}

// order reduces crossings with barycenter sweeps, keeping the best result.
func (l *layout) order() {
    index := func() {
        for _, rank := range l.ranks {
            for i, n := range rank {
                n.order = i
            }
        }
    }
    index()
    best, bestCross := l.snapshot(), l.crossings()
    sweep := func(r int, neighbours map[*fnode][]*fnode) {
        rank := l.ranks[r]
        weight := map[*fnode]float64{}
        for _, n := range rank {
            weight[n] = float64(n.order)
            if ns := neighbours[n]; len(ns) > 0 {
                sum := 0
                for _, m := range ns {
                    sum += m.order
                }
                weight[n] = float64(sum) / float64(len(ns))
            }
        }
        sort.SliceStable(rank, func(i, j int) bool { return weight[rank[i]] < weight[rank[j]] })
        for i, n := range rank {
            n.order = i
        }
    }
    for iter := 0; iter < 6 && bestCross > 0; iter++ {
        for r := 1; r < len(l.ranks); r++ {
            sweep(r, l.preds)
        }
        for r := len(l.ranks) - 2; r >= 0; r-- {
            sweep(r, l.succs)
        }
        if c := l.crossings(); c < bestCross {
            best, bestCross = l.snapshot(), c
        }
    }
    l.ranks = best
    index()
}

func (l *layout) snapshot() [][]*fnode {
    out := make([][]*fnode, len(l.ranks))
    for i, r := range l.ranks {
        out[i] = append([]*fnode{}, r...)
    }
    return out
}
func (n *fnode) boxSize() (w, h int) {
    if n.dummy {
        return 1, 1
    }
    if n.loop && len(n.label) > 0 {
        n.label[len(n.label)-1] += " ↺"
        n.loop = false
    }
    for _, s := range n.label {
        w = max(w, utf8.RuneCountInString(s))
    }
    if n.shape == shapeBare {
        return max(w, 1), 1
    }
    w += 4
    if w%2 == 0 {
        w++ // odd widths give edges a true centre column
    }
    return w, len(n.label) + 2
}

func (l *layout) measure() {
    for _, rank := range l.ranks {
        for _, n := range rank {
            w, h := n.boxSize()
            if l.vertical {
                n.sizeO, n.sizeR = w, h
            } else {
                n.sizeO, n.sizeR = h, w
            }
        }
    }
}

// placeO positions nodes along the order axis: left to right in rank order,
// then nudged right toward the average of their neighbours so parents sit
// over children and chains run straight.
func (l *layout) placeO() {
    gap := 1
    if l.vertical {
        gap = 3
    }
    sep := func(a, b *fnode) int {
        if a.dummy && b.dummy {
            return max(gap-1, 1)
        }
        return gap
    }
    centre := func(ns []*fnode) int {
        sum := 0
        for _, m := range ns {
            sum += m.centerO()
        }
        return sum / len(ns)
    }
    for _, rank := range l.ranks {
        for i, n := range rank {
            at := 0
            if i > 0 {
                at = rank[i-1].posO + rank[i-1].sizeO + sep(rank[i-1], n)
            }
            if ps := l.preds[n]; len(ps) > 0 {
                want := centre(ps) - n.sizeO/2
                if i == 0 || want > at {
                    at = want
                }
            }
            n.posO = at
        }
    }
    nudge := func(rank []*fnode, neighbours map[*fnode][]*fnode) {
        for i := len(rank) - 1; i >= 0; i-- {
            n := rank[i]
            ns := neighbours[n]
            if len(ns) == 0 {
                continue
            }
            want := centre(ns) - n.sizeO/2
            if i+1 < len(rank) {
                want = min(want, rank[i+1].posO-sep(n, rank[i+1])-n.sizeO)
            }
            n.posO = max(n.posO, want)
        }
    }
    for iter := 0; iter < 3; iter++ {
        for r := len(l.ranks) - 2; r >= 0; r-- {
            nudge(l.ranks[r], l.succs)
        }
        for r := 1; r < len(l.ranks); r++ {
            nudge(l.ranks[r], l.preds)
        }
    }

    // Normalise, leaving room on the left for labels centred on an edge.
    lo := 1 << 30
    for _, rank := range l.ranks {
        for _, n := range rank {
            lo = min(lo, n.posO)
        }
    }
    if l.vertical {
        for _, segs := range l.segs {
            for _, s := range segs {
                if !s.last || s.edge.label == "" {
                    continue
                }
                n := utf8.RuneCountInString(s.edge.label)
                if port := l.portTo(s); !s.edge.reversed {
                    lo = min(lo, port-n/2)
                } else if port < s.to.centerO() {
                    lo = min(lo, port-1-n)
                }
            }
        }
    }
    for _, rank := range l.ranks {
        for _, n := range rank {
            n.posO -= lo
        }
    }
}

An edge leaves a node from the middle of its side, runs to a track in the gap between two ranks, follows the track, and then heads for its target. All the edges from one source share a track, so they look like a tree. Different sources get different tracks, so edges only merge where they really merge.

// Back edges get their own port, off-centre, so they neither share a trunk
// with forward edges nor merge into a forward edge's arrowhead.
func (l *layout) portOffset(n *fnode, toward int) int {
    if !l.vertical || n.dummy || n.shape == shapeBare {
        return 0
    }
    off := min(2, (n.sizeO-3)/2)
    if toward < n.centerO() {
        return -off // leave on the side the edge is heading for
    }
    return off
}

func (l *layout) portFrom(s *seg) int {
    if s.first && s.edge.reversed {
        return s.from.centerO() + l.portOffset(s.from, s.to.centerO())
    }
    return s.from.centerO()
}

func (l *layout) portTo(s *seg) int {
    if s.last && s.edge.reversed {
        return s.to.centerO() + l.portOffset(s.to, s.from.centerO())
    }
    return s.to.centerO()
}

// tracks gives each source in a gap its own lane for the run across the
// order axis, so edges from different sources never merge by accident.
func (l *layout) tracks() {
    n := len(l.segs)
    l.gapStub, l.gapTracks, l.gapLabel = make([]int, n), make([]int, n), make([]int, n)
    for g, segs := range l.segs {
        type group struct {
            lo, hi int
            segs   []*seg
        }
        type key struct {
            n    *fnode
            port int
        }
        bySource := map[key]*group{}
        var groups []*group
        for _, s := range segs {
            k := key{s.from, l.portFrom(s)}
            gr := bySource[k]
            if gr == nil {
                gr = &group{lo: k.port, hi: k.port}
                bySource[k] = gr
                groups = append(groups, gr)
            }
            gr.lo, gr.hi = min(gr.lo, l.portTo(s)), max(gr.hi, l.portTo(s))
            gr.segs = append(gr.segs, s)

            if s.last && s.edge.label != "" {
                if l.vertical {
                    l.gapLabel[g] = 1
                } else {
                    l.gapLabel[g] = max(l.gapLabel[g], utf8.RuneCountInString(s.edge.label)+2)
                }
            }
            if !l.vertical || (s.first && (s.edge.reversed || s.edge.tailHead)) {
                l.gapStub[g] = 1
            }
        }
        sort.SliceStable(groups, func(i, j int) bool { return groups[i].lo < groups[j].lo })
        var lanes [][]*group
        for _, gr := range groups {
            if gr.lo == gr.hi {
                continue // straight through: needs no lane of its own
            }
            placed := false
            for t := range lanes {
                free := true
                for _, other := range lanes[t] {
                    if gr.lo <= other.hi+1 && other.lo <= gr.hi+1 {
                        free = false
                        break
                    }
                }
                if free {
                    lanes[t] = append(lanes[t], gr)
                    for _, s := range gr.segs {
                        s.track = t
                    }
                    placed = true
                    break
                }
            }
            if !placed {
                lanes = append(lanes, []*group{gr})
                for _, s := range gr.segs {
                    s.track = len(lanes) - 1
                }
            }
        }
        l.gapTracks[g] = max(len(lanes), 1)
    }
}

func (l *layout) trackStep() int {
    if l.vertical {
        return 1
    }
    return 2
}

func (l *layout) placeR() {
    l.rankStart, l.rankSize = make([]int, len(l.ranks)), make([]int, len(l.ranks))
    at := 0
    for r, rank := range l.ranks {
        for _, n := range rank {
            l.rankSize[r] = max(l.rankSize[r], n.sizeR)
        }
        l.rankStart[r] = at
        for _, n := range rank {
            n.posR = at
            if n.dummy {
                n.sizeR = l.rankSize[r]
            }
        }
        at += l.rankSize[r]
        if r < len(l.segs) {
            at += l.gapStub[r] + l.gapTracks[r]*l.trackStep() + l.gapLabel[r] + 1
        }
    }
    l.totalR = at
}

// trackPos is the rank-axis coordinate of a segment's lane.
func (l *layout) trackPos(gap int, s *seg) int {
    return l.rankStart[gap] + l.rankSize[gap] + l.gapStub[gap] + s.track*l.trackStep()
}

Drawing

internal/mermaid/draw.go
// pt converts layout coordinates (rank axis, order axis) to canvas x, y.
func (l *layout) pt(r, o int) (x, y int) {
    if l.mirror {
        r = l.totalR - 1 - r
    }
    if l.vertical {
        return o, r
    }
    return r, o
}

func (l *layout) stroke(c *canvas, r1, o1, r2, o2 int, st lineStyle) {
    x1, y1 := l.pt(r1, o1)
    x2, y2 := l.pt(r2, o2)
    c.stroke(x1, y1, x2, y2, st)
}

// arrow picks the head glyph for travel from (r1,o1) toward (r2,o2).
func (l *layout) arrow(kind rune, r1, o1, r2, o2 int) rune {
    switch kind {
    case 'x':
        return '×'
    case 'o':
        return '○'
    }
    x1, y1 := l.pt(r1, o1)
    x2, y2 := l.pt(r2, o2)
    switch {
    case x2 > x1:
        return '▶'
    case x2 < x1:
        return '◀'
    case y2 < y1:
        return '▲'
    }
    return '▼'
}

func (l *layout) draw() []string {
    c := &canvas{}
    type labelAt struct {
        x, y int
        s    string
    }
    var labels []labelAt
    var heads []labelAt

    for g, segs := range l.segs {
        for _, s := range segs {
            e := s.edge
            o1, o2 := l.portFrom(s), l.portTo(s)
            start := s.from.posR + s.from.sizeR - 1
            track := l.trackPos(g, s)
            end := s.to.posR // the target's border (or the dummy's first cell)

            // An arrowhead sits just outside the box it points at; the line
            // stops there. Open ends run on into the border.
            headAtEnd := s.last && !e.reversed && e.head != 0 || s.last && e.reversed && e.tailHead
            headAtStart := s.first && e.reversed && e.head != 0 || s.first && !e.reversed && e.tailHead
            if s.to.shape == shapeBare && !s.to.dummy {
                headAtEnd = false
                end--
            }
            stop := end
            if headAtEnd {
                stop = end - 1
            }
            l.stroke(c, start, o1, track, o1, e.style)
            l.stroke(c, track, o1, track, o2, e.style)
            l.stroke(c, track, o2, stop, o2, e.style)

            if headAtEnd {
                kind := e.head
                if e.reversed {
                    kind = '>'
                }
                x, y := l.pt(stop, o2)
                heads = append(heads, labelAt{x, y, string(l.arrow(kind, track, o2, end, o2))})
            }
            if headAtStart {
                kind := e.head
                if !e.reversed {
                    kind = '>'
                }
                x, y := l.pt(start+1, o1)
                heads = append(heads, labelAt{x, y, string(l.arrow(kind, track, o1, start, o1))})
            }
            if s.last && e.label != "" {
                n := utf8.RuneCountInString(e.label)
                if l.vertical {
                    x, y := l.pt(end-2, o2)
                    if e.reversed && o2 < s.to.centerO() {
                        labels = append(labels, labelAt{x - 1 - n, y, e.label})
                    } else if e.reversed {
                        labels = append(labels, labelAt{x + 2, y, e.label})
                    } else {
                        labels = append(labels, labelAt{x - n/2, y, e.label})
                    }
                } else {
                    zone := l.gapLabel[g]
                    r := end - 1 - zone + (zone-n)/2
                    if l.mirror {
                        r += n - 1
                    }
                    x, y := l.pt(r, o2)
                    labels = append(labels, labelAt{x, y, e.label})
                }
            }
        }
    }

    for _, rank := range l.ranks {
        for _, n := range rank {
            x1, y1 := l.pt(n.posR, n.posO)
            x2, y2 := l.pt(n.posR+n.sizeR-1, n.posO+n.sizeO-1)
            x, y := min(x1, x2), min(y1, y2)
            w, h := abs(x2-x1)+1, abs(y2-y1)+1
            if n.dummy {
                l.stroke(c, n.posR, n.centerO(), n.posR+n.sizeR-1, n.centerO(), solid)
                continue
            }
            if n.shape == shapeBare {
                c.write(x, y, n.label[0])
                continue
            }
            // Blank the interior so nothing shows through the box.
            for yy := y + 1; yy < y+h-1; yy++ {
                for xx := x + 1; xx < x+w-1; xx++ {
                    *c.at(xx, yy) = cell{r: ' '}
                }
            }
            st := map[shape]lineStyle{shapeRound: rounded, shapeDouble: double}[n.shape]
            c.rect(x, y, w, h, st)
            if n.shape == shapeDiamond {
                for _, p := range [][2]int{{x, y}, {x + w - 1, y}, {x, y + h - 1}, {x + w - 1, y + h - 1}} {
                    c.put(p[0], p[1], '◆')
                }
            }
            for i, s := range n.label {
                c.write(x+(w-utf8.RuneCountInString(s))/2, y+1+i, s)
            }
        }
    }
    for _, h := range heads {
        c.write(h.x, h.y, h.s)
    }
    for _, lb := range labels {
        c.write(lb.x, lb.y, lb.s)
    }
    return c.lines()
}

func abs(n int) int {
    if n < 0 {
        return -n
    }
    return n
}

Sequence diagrams

internal/mermaid/sequence.go

Participants go across the top and time runs down. The columns are spaced so every message label fits between the lifelines it spans.

type participant struct {
    id, label string
    x         int // lifeline column
    w         int // header box width
}

type seqEvent struct {
    kind     string // message, note, block
    from, to *participant
    text     string
    dotted   bool
    head     string // ">" filled, ")" async, "x" cross, "" open
    side     string // notes: over, left, right
}

var (
    participantRe = regexp.MustCompile(`^(?:participant|actor)\s+(\S+?)(?:\s+as\s+(.+))?$`)
    messageRe     = regexp.MustCompile(`^(.+?)\s*(<<-->>|<<->>|-->>|->>|--x|-x|--\)|-\)|-->|->)\s*([+-]?)\s*([^:]+?)\s*:\s*(.*)$`)
    noteRe        = regexp.MustCompile(`(?i)^note\s+(over|left of|right of)\s+([^:]+?)\s*:\s*(.*)$`)
    blockRe       = regexp.MustCompile(`^(loop|alt|opt|par|critical|break|else|and|option|rect)\b\s*(.*)$`)
)

func rlen(s string) int { return utf8.RuneCountInString(s) }

func renderSequence(lines []string) ([]string, error) {
    var parts []*participant
    byID := map[string]*participant{}
    get := func(id string) *participant {
        id = strings.TrimSpace(id)
        if p, ok := byID[id]; ok {
            return p
        }
        p := &participant{id: id, label: id}
        byID[id] = p
        parts = append(parts, p)
        return p
    }

    var events []seqEvent
    for _, s := range lines {
        switch {
        case participantRe.MatchString(s):
            m := participantRe.FindStringSubmatch(s)
            p := get(m[1])
            if m[2] != "" {
                p.label = strings.Join(cleanLabel(m[2]), " ")
            }
        case noteRe.MatchString(s):
            m := noteRe.FindStringSubmatch(s)
            ids := strings.Split(m[2], ",")
            ev := seqEvent{kind: "note", side: strings.ToLower(strings.Fields(m[1])[0]), from: get(ids[0]), text: strings.Join(cleanLabel(m[3]), " ")}
            ev.to = ev.from
            if len(ids) > 1 {
                ev.to = get(ids[1])
            }
            events = append(events, ev)
        case s == "end":
            events = append(events, seqEvent{kind: "block", text: "end"})
        case blockRe.MatchString(s):
            m := blockRe.FindStringSubmatch(s)
            label := m[1]
            if m[2] != "" && m[1] != "rect" {
                label += ": " + m[2]
            }
            events = append(events, seqEvent{kind: "block", text: label})
        case messageRe.MatchString(s):
            m := messageRe.FindStringSubmatch(s)
            ev := seqEvent{kind: "message", from: get(m[1]), to: get(m[4]), text: strings.Join(cleanLabel(m[5]), " ")}
            arrow := m[2]
            ev.dotted = strings.HasPrefix(arrow, "--") || strings.HasPrefix(arrow, "<<--")
            switch {
            case strings.HasSuffix(arrow, ">>"):
                ev.head = ">"
            case strings.HasSuffix(arrow, "x"):
                ev.head = "x"
            case strings.HasSuffix(arrow, ")"):
                ev.head = ")"
            }
            events = append(events, ev)
        }
    }
    if len(parts) == 0 {
        return nil, fmt.Errorf("sequence diagram has no participants")
    }

    // Column spacing: wide enough for headers and for every label that has to
    // fit between two lifelines.
    index := map[*participant]int{}
    for i, p := range parts {
        index[p] = i
        p.w = rlen(p.label) + 4
        if p.w%2 == 0 {
            p.w++
        }
    }
    gaps := make([]int, len(parts)) // gaps[i]: distance from lifeline i to i+1
    for i := 0; i+1 < len(parts); i++ {
        gaps[i] = parts[i].w/2 + parts[i+1].w/2 + 3
    }
    tail := 0 // room needed right of the last lifeline
    for _, ev := range events {
        if ev.from == nil {
            continue
        }
        a, b := index[ev.from], index[ev.to]
        if a > b {
            a, b = b, a
        }
        need := rlen(ev.text) + 4
        switch {
        case ev.kind == "note" && ev.side == "right", a == b && ev.kind == "message":
            need = rlen(ev.text) + 6
            if a+1 < len(parts) {
                gaps[a] = max(gaps[a], need)
            } else {
                tail = max(tail, need)
            }
        case ev.kind == "note" && ev.side == "left":
            if a > 0 {
                gaps[a-1] = max(gaps[a-1], need+2)
            }
        case a != b:
            have := 0
            for i := a; i < b; i++ {
                have += gaps[i]
            }
            if extra := need - have; extra > 0 {
                per := (extra + (b - a) - 1) / (b - a)
                for i := a; i < b; i++ {
                    gaps[i] += per
                }
            }
        }
    }
    x := parts[0].w / 2
    for _, ev := range events {
        if ev.kind == "note" && ev.side == "left" && index[ev.from] == 0 {
            x = max(x, rlen(ev.text)+6)
        }
    }
    for i, p := range parts {
        p.x = x
        x += gaps[i]
    }
    last := parts[len(parts)-1]
    totalW := max(last.x+last.w/2+1, last.x+tail)

    c := &canvas{}
    for _, p := range parts {
        c.rect(p.x-p.w/2, 0, p.w, 3, solid)
        c.write(p.x-rlen(p.label)/2, 1, p.label)
    }
    y := 3
    lifelines := func(rows int) {
        for i := 0; i < rows; i++ {
            for _, p := range parts {
                c.stroke(p.x, y-1, p.x, y, solid)
            }
            y++
        }
    }
    for _, ev := range events {
        switch ev.kind {
        case "block":
            lifelines(1)
            label := "[" + ev.text + "]"
            for xx := 0; xx < totalW; xx++ {
                *c.at(xx, y-1) = cell{r: '┄'}
            }
            c.write(2, y-1, " "+label+" ")
        case "note":
            a, b := ev.from, ev.to
            if index[a] > index[b] {
                a, b = b, a
            }
            w := rlen(ev.text) + 4
            var left int
            switch ev.side {
            case "right":
                left = a.x + 2
            case "left":
                left = a.x - 1 - w
            default:
                span := b.x - a.x
                w = max(w, span+5)
                left = (a.x+b.x)/2 - w/2
            }
            left = max(left, 0)
            lifelines(3)
            for yy := y - 3; yy < y; yy++ {
                for xx := left; xx < left+w; xx++ {
                    *c.at(xx, yy) = cell{}
                }
            }
            c.rect(left, y-3, w, 3, rounded)
            c.write(left+(w-rlen(ev.text))/2, y-2, ev.text)
        case "message":
            if ev.from == ev.to {
                lifelines(3)
                px := ev.from.x
                c.write(px+2, y-3, ev.text)
                c.stroke(px, y-2, px+4, y-2, solid)
                c.stroke(px+4, y-2, px+4, y-1, rounded)
                c.stroke(px+4, y-1, px+2, y-1, solid)
                c.at(px+4, y-2).style, c.at(px+4, y-1).style = rounded, rounded
                c.put(px+1, y-1, '◀')
                continue
            }
            lifelines(2)
            ax, bx := ev.from.x, ev.to.x
            dir := sign(bx - ax)
            lo, hi := min(ax, bx), max(ax, bx)
            c.write(lo+1+(hi-lo-1-rlen(ev.text))/2, y-2, ev.text)
            st := solid
            if ev.dotted {
                st = dotted
            }
            c.stroke(ax, y-1, bx-dir, y-1, st)
            head := map[string][2]rune{">": {'▶', '◀'}, ")": {'▷', '◁'}, "x": {'×', '×'}}[ev.head]
            if head[0] != 0 {
                if dir > 0 {
                    c.put(bx-1, y-1, head[0])
                } else {
                    c.put(bx+1, y-1, head[1])
                }
            }
        }
    }
    lifelines(1)
    return c.lines(), nil
}
internal/mermaid/mermaid_test.go
var samples = []string{
    `graph TD
    A[Spawn n workers] --> B[Feed m jobs]
    B --> C{ctx cancelled?}
    C -->|yes| D[Return partial]
    C -->|no| E[Collect results]
    D --> F(Done)
    E --> F`,
    `flowchart LR
    jobs[(jobs chan)] --> w1[worker 1] & w2[worker 2] & w3[worker 3]
    w1 & w2 & w3 --> results[(results chan)]
    results -- collect --> main`,
    `graph TD
    A --> B --> C
    C -->|retry| A
    A --> D
    B -.-> D
    D ==> D`,
    `graph BT
    leaf --> root`,
    `stateDiagram-v2
    [*] --> Idle
    Idle --> Running: start
    Running --> Idle: stop
    Running --> [*]`,
    `sequenceDiagram
    participant M as main
    participant W as worker
    M->>W: job
    Note right of W: job * 2
    W-->>M: result
    loop until closed
      M->>M: collect
    end
    M-xW: cancel`,
}

func TestRenderSamples(t *testing.T) {
    for _, src := range samples {
        out, err := Render(src)
        if err != nil {
            t.Fatalf("%v\n%s", err, src)
        }
        t.Logf("\n%s\n\n%s", src, strings.Join(out, "\n"))
    }
}

func TestUnsupportedAndMalformed(t *testing.T) {
    if _, err := Render("gantt\n title x"); err == nil {
        t.Error("expected an error for gantt")
    }
    for _, src := range []string{"", "graph", "graph TD\n-->", "graph TD\nA -->", "sequenceDiagram", "stateDiagram-v2", "graph LR\nA[unclosed --> B"} {
        Render(src) // must not panic
    }
}

Weaving

internal/weave/weave.go

Weaving turns the document into something you read: a PDF typeset by Typst, or an HTML page. pandoc does the Markdown conversion and translates the TeX math into whatever the output wants. litgo adds what the source hides. The chunk: and file: directives are HTML comments, invisible on the page, so they become headings over their blocks, with links from each chunk to the places that use it and from each reference back to its chunk.

flowchart LR
    doc[name.lit.md] --> woven[woven Markdown] --> pandoc
    pandoc --> typ[Typst markup] --> typst --> pdf[name.pdf]
    pandoc --> html[name.html]

Each format has its own target, so it can decide things like escaping for itself.

var slugRe = regexp.MustCompile(`[^a-z0-9]+`)

// slug is the anchor of a chunk, the same in every format.
func slug(name string) string {
    return "chunk-" + strings.Trim(slugRe.ReplaceAllString(strings.ToLower(name), "-"), "-")
}

// use is a place a chunk is referenced from. An empty ID is the program
// itself: an unnamed block.
type use struct{ ID, Name string }

// target is how one output format writes the things litgo adds.
type target struct {
    chunk     func(id, name, sign string, uses []use) string
    file      func(path string) string
    meta      func(text string) string // goes under the title; nil leaves it out
    literal   [2]string                // put around blocks whose references are only text
    diagrams  bool                     // draw Mermaid here: the format cannot run mermaid.js
    figure    func(svg string) string  // places a diagram that mermaid.js drew
    figures   map[int]string           // those drawings, by the line their block opens on
    dropTitle bool                     // the format sets the title itself
}

The woven Markdown is the source, with some lines added before, after, or in place of a few of its lines.

func weave(d *lit.Doc, t target) string {
    before, replace, after := map[int]string{}, map[int]string{}, map[int]string{}
    until := map[int]int{} // a replacement at line i stands for lines i to until[i]

    // <<label every chunk with its name and its uses>>
    for _, f := range d.Files[1:] {
        replace[f.Line] = t.file(f.Path)
    }
    // <<draw the diagrams if the format cannot>>
    // <<mark the blocks whose references are only text>>
    // <<put the summary under the title>>

    var sb strings.Builder
    for i := 0; i < len(d.Lines); i++ {
        sb.WriteString(before[i])
        if r, ok := replace[i]; ok {
            sb.WriteString(r)
            i = max(i, until[i])
            continue
        }
        sb.WriteString(d.Lines[i] + "\n")
        sb.WriteString(after[i])
    }
    return sb.String()
}

Only the first block of a chunk gets the anchor. Later blocks with the same name continue the chunk, and the sign says so, in the notation literate programs have used since WEB.

⟨label every chunk with its name and its uses⟩ ≡ used in the program
usedIn := map[string][]use{}
for _, c := range d.Chunks() {
    seen := map[string]bool{}
    for _, r := range c.Refs {
        u := use{Name: "the program"}
        if b := d.BlockAt(r.Line); b != nil && b.Chunk != "" {
            u = use{slug(b.Chunk), b.Chunk}
        }
        if !seen[u.Name] {
            seen[u.Name] = true
            usedIn[c.Name] = append(usedIn[c.Name], u)
        }
    }
}
count := map[string]int{}
for _, b := range d.Blocks {
    if b.Chunk == "" || !b.Tangled() {
        continue
    }
    count[b.Chunk]++
    id, sign := slug(b.Chunk), "≡"
    if count[b.Chunk] > 1 {
        id, sign = "", "+≡"
    }
    replace[b.ChunkLine] = t.chunk(id, b.Chunk, sign, usedIn[b.Chunk])
}

A web page can load mermaid.js. A PDF can’t, so the diagrams have to be drawn before Typst ever sees them. If mermaid.js drew a block, its SVG goes in as a figure. If not, litgo already knows how to draw a diagram with box characters, and in a monospaced font that still makes a decent figure. If litgo can’t draw it either, it stays as source.

⟨draw the diagrams if the format cannot⟩ ≡ used in the program
if t.diagrams {
    for _, b := range d.Blocks {
        if b.Lang != "mermaid" || b.Strip > 0 {
            continue
        }
        if svg, ok := t.figures[b.Open]; ok {
            replace[b.Open], until[b.Open] = t.figure(svg), b.Close
            continue
        }
        lines, err := mermaid.Render(strings.Join(d.Lines[b.First():b.Last()+1], "\n"))
        if err != nil {
            continue
        }
        replace[b.Open], until[b.Open] = "```diagram\n"+strings.Join(lines, "\n")+"\n```\n", b.Close
    }
}

The drawing is done by mermaid-cli, which runs mermaid.js in a headless browser. Starting a browser is slow, so every diagram goes to mmdc in one trip, as a Markdown file of nothing but Mermaid blocks. It writes out-1.svg, out-2.svg and so on beside the name it’s given. The catch: one diagram with a mistake fails the whole batch. When that happens each diagram gets a run of its own, so only the broken one is drawn as text. Without mmdc, they all are.

Typst draws SVG text itself, but not the HTML that mermaid.js likes to stuff into labels, so htmlLabels is off.

// drawn returns the SVG of each Mermaid block that mermaid-cli could draw,
// by the line the block opens on.
func drawn(d *lit.Doc) map[int]string {
    var at []int
    var srcs []string
    for _, b := range d.Blocks {
        if b.Lang == "mermaid" && b.Strip == 0 {
            at = append(at, b.Open)
            srcs = append(srcs, strings.Join(d.Lines[b.First():b.Last()+1], "\n"))
        }
    }
    if len(srcs) == 0 {
        return nil
    }
    bin, err := exec.LookPath("mmdc")
    if err != nil {
        fmt.Fprintln(os.Stderr, "litgo: mmdc is not installed (npm install -g @mermaid-js/mermaid-cli), so diagrams are drawn as text")
        return nil
    }
    svgs := mmdc(bin, srcs)
    if svgs == nil && len(srcs) > 1 {
        svgs = make([]string, len(srcs))
        for i, src := range srcs {
            if one := mmdc(bin, []string{src}); one != nil {
                svgs[i] = one[0]
            }
        }
    }
    figures := map[int]string{}
    for i, svg := range svgs {
        if svg != "" {
            figures[at[i]] = svg
        }
    }
    return figures
}

// mmdc draws every diagram in one run of mermaid-cli, or returns nil.
func mmdc(bin string, srcs []string) []string {
    dir, err := os.MkdirTemp("", "litgo-mermaid")
    if err != nil {
        return nil
    }
    defer os.RemoveAll(dir)
    var md strings.Builder
    for _, src := range srcs {
        md.WriteString("```mermaid\n" + src + "\n```\n\n")
    }
    os.WriteFile(filepath.Join(dir, "in.md"), []byte(md.String()), 0o644)
    os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"htmlLabels": false, "flowchart": {"htmlLabels": false}}`), 0o644)
    cmd := exec.Command(bin, "-i", "in.md", "-o", "out.svg", "-c", "config.json", "-b", "transparent", "-q")
    cmd.Dir = dir
    if err := cmd.Run(); err != nil {
        return nil
    }
    svgs := make([]string, len(srcs))
    for i := range srcs {
        svg, err := os.ReadFile(filepath.Join(dir, fmt.Sprintf("out-%d.svg", i+1)))
        if err != nil {
            return nil
        }
        svgs[i] = string(svg)
    }
    return svgs
}

References in woven code show up as ⟨names⟩, with links. Text that only looks like a reference shouldn’t get that treatment, which means verbatim blocks and illustrations that aren’t tangled at all.

⟨mark the blocks whose references are only text⟩ ≡ used in the program
if t.literal[0] != "" {
    for _, b := range d.Blocks {
        if (b.Tangled() && !b.Verbatim) || b.Strip > 0 {
            continue
        }
        if _, drawn := replace[b.Open]; drawn {
            continue
        }
        for i := b.First(); i <= b.Last(); i++ {
            if len(lit.RefsIn(d.Lines[i], i)) > 0 {
                before[b.Open] += t.literal[0]
                after[min(b.Close, len(d.Lines)-1)] += t.literal[1]
                break
            }
        }
    }
}
⟨put the summary under the title⟩ ≡ used in the program
title := titleLine(d)
if title >= 0 && t.dropTitle {
    replace[title] = ""
}
if t.meta != nil {
    if title >= 0 {
        after[title] += "\n" + t.meta(Summary(d))
    } else {
        before[0] = t.meta(Summary(d)) + "\n" + before[0]
    }
}
// titleLine is the line of the first top-level heading, or -1.
func titleLine(d *lit.Doc) int {
    for i, l := range d.Lines {
        if strings.HasPrefix(l, "# ") && d.BlockAt(i) == nil {
            return i
        }
    }
    return -1
}

// Title is the first top-level heading, or the file name.
func Title(d *lit.Doc) string {
    if i := titleLine(d); i >= 0 {
        return strings.TrimSpace(d.Lines[i][2:])
    }
    return filepath.Base(d.Path)
}

// Summary says in a line what the document tangles to.
func Summary(d *lit.Doc) string {
    def := d.Files[0]
    if len(d.Files) > 1 {
        return fmt.Sprintf("tangles to %d files", len(d.Tangle().Files))
    }
    var parts []string
    if def.Package != "" {
        parts = append(parts, "package "+def.Package)
    }
    if len(def.Imports) > 0 {
        var names []string
        for _, imp := range def.Imports {
            names = append(names, imp.Path)
        }
        parts = append(parts, "imports "+strings.Join(names, ", "))
    }
    parts = append(parts, "tangles to "+filepath.Base(d.OutPath(def)))
    return strings.Join(parts, " · ")
}

Both formats go through pandoc, and pandoc reads the Markdown the same way for both.

func pandoc(markdown string, args ...string) ([]byte, error) {
    bin, err := exec.LookPath("pandoc")
    if err != nil {
        return nil, fmt.Errorf("pandoc is not installed; `litgo weave --to md` needs nothing, everything else needs pandoc")
    }
    args = append([]string{"--from", "markdown+tex_math_dollars+tex_math_single_backslash-implicit_figures"}, args...)
    cmd := exec.Command(bin, args...)
    cmd.Stdin = strings.NewReader(markdown)
    var stderr strings.Builder
    cmd.Stderr = &stderr
    out, err := cmd.Output()
    if err != nil {
        return nil, fmt.Errorf("pandoc: %v\n%s", err, stderr.String())
    }
    return out, nil
}

HTML

var htmlTarget = target{
    chunk: func(id, name, sign string, uses []use) string {
        if id != "" {
            id = ` id="` + id + `"`
        }
        var places []string
        for _, u := range uses {
            if u.ID == "" {
                places = append(places, u.Name)
            } else {
                places = append(places, fmt.Sprintf(`<a href="#%s">⟨%s⟩</a>`, u.ID, html.EscapeString(u.Name)))
            }
        }
        where := "never used"
        if len(places) > 0 {
            where = "used in " + strings.Join(places, ", ")
        }
        return fmt.Sprintf(`<div class="lit-chunk"%s><span class="lit-name">⟨%s⟩ %s</span> <span class="lit-uses">%s</span></div>`+"\n",
            id, html.EscapeString(name), sign, where)
    },
    file: func(path string) string {
        return `<div class="lit-file">` + html.EscapeString(path) + "</div>\n"
    },
    meta: func(text string) string {
        return `<div class="lit-meta">` + html.EscapeString(text) + "</div>\n"
    },
}

// Markdown returns the woven Markdown, with litgo's additions as HTML.
func Markdown(d *lit.Doc) string { return weave(d, htmlTarget) }

KaTeX typesets the math and mermaid.js draws the diagrams. pandoc writes a table of contents from the # and ## headings, and where it goes depends on the width of the window. On a wide screen it’s a column down the left, pinned in place while the text scrolls. On a narrow one it folds away behind a button in the corner and opens over the text when asked. That part is all stylesheet: the button is a label for a checkbox that pandoc puts in front of the contents, and the checkbox’s state picks the rules. The script does the two things a stylesheet can’t. It drops the title’s own entry when the title is the only #, so the contents start with what comes after, and it marks the section the reader is in as the page scrolls. It leaves the history alone, so a link in the contents is just a link, and the back button undoes it. Markdown can’t link references inside highlighted code, so a few more lines turn the comments that hold references into links once the page loads.

// HTML weaves a standalone page.
func HTML(d *lit.Doc, out string) error {
    tmp, err := os.MkdirTemp("", "litgo-weave")
    if err != nil {
        return err
    }
    defer os.RemoveAll(tmp)
    header := filepath.Join(tmp, "header.html")
    if err := os.WriteFile(header, []byte(headerHTML), 0o644); err != nil {
        return err
    }
    toggle := filepath.Join(tmp, "toggle.html")
    if err := os.WriteFile(toggle, []byte(toggleHTML), 0o644); err != nil {
        return err
    }
    _, err = pandoc(Markdown(d), "--to", "html5", "--standalone", "--katex",
        "--toc", "--toc-depth=2", "--variable", "toc-title=Contents",
        "--metadata", "pagetitle="+Title(d), "--include-in-header", header,
        "--include-before-body", toggle, "--output", out)
    return err
}

// toggleHTML precedes the table of contents in the body, so that the
// stylesheet can reach the contents from the checkbox with a sibling selector.
const toggleHTML = `<input type="checkbox" id="toc-toggle">
<label for="toc-toggle"><span class="toc-open">Contents</span><span class="toc-close">Close</span></label>
`

const headerHTML = `<style>
:root { --fg:#1f2328; --bg:#ffffff; --muted:#656d76; --accent:#0a58ca; --code-bg:#f6f8fa; --rule:#d0d7de; }
@media (prefers-color-scheme: dark) {
  :root { --fg:#e6edf3; --bg:#0d1117; --muted:#8b949e; --accent:#79b8ff; --code-bg:#161b22; --rule:#30363d; }
}
html { color: var(--fg); background: var(--bg); }
body { max-width: 49rem; margin: 0 auto; padding: 2.5rem 1rem 6rem; font: 17px/1.6 Georgia, "Iowan Old Style", serif; }
h1, h2, h3 { font-family: system-ui, sans-serif; line-height: 1.25; }
h2 { margin-top: 2.2em; border-bottom: 1px solid var(--rule); padding-bottom: .2em; }
a { color: var(--accent); text-decoration: none; } a:hover { text-decoration: underline; }
code { font: 0.86em/1.5 "JetBrains Mono", ui-monospace, Menlo, monospace; }
pre, div.sourceCode { background: var(--code-bg); border-radius: 6px; }
pre { padding: .8em 1em; overflow-x: auto; margin: 0; }
div.sourceCode { margin: 0 0 1.2em; }
.lit-meta { color: var(--muted); font: 14px system-ui, sans-serif; margin: -0.6em 0 2em; }
.lit-chunk { font: 15px system-ui, sans-serif; margin: 1.4em 0 .3em; display: flex; justify-content: space-between; gap: 1em; align-items: baseline; }
.lit-name { font-weight: 600; font-style: italic; }
.lit-file { font: 600 13px "JetBrains Mono", ui-monospace, monospace; color: var(--muted); border-top: 1px solid var(--rule); margin: 2.4em 0 1em; padding-top: .5em; }
.lit-file::before { content: "▸ "; }
.lit-uses { color: var(--muted); font-size: 13px; }
a.lit-ref { font-style: italic; font-family: Georgia, serif; }
:target { scroll-margin-top: 1em; } .lit-chunk:target .lit-name { background: color-mix(in srgb, var(--accent) 22%, transparent); }
pre.mermaid { background: transparent; text-align: center; }
/* Dark mode brightens pandoc's highlighting. It used to do so with a CSS filter
   on every span, but a filter gives each of the tens of thousands of spans a
   layer of its own: some 350MB more in WebKit, which iOS Safari doesn't have,
   so it gave up and reloaded the page for ever. These are the colours that
   filter produced, written down once. */
@media (prefers-color-scheme: dark) {
  code span.al, code span.er { color: #da1d1d; }
  code span.an, code span.co, code span.cv, code span.in, code span.wa { color: #b2fbfb; }
  code span.at { color: #d7f165; }
  code span.bn, code span.dv, code span.fl { color: #83f8c4; }
  code span.bu, code span.im { color: #1fcd1f; }
  code span.cf, code span.kw, code span.ot { color: #1cb448; }
  code span.ch, code span.sc, code span.st, code span.vs { color: #7bbcf0; }
  code span.cn { color: #d91b1b; }
  code span.do { color: #e87474; }
  code span.dt { color: #e66e33; }
  code span.fu { color: #3372e2; }
  code span.op { color: #adadad; }
  code span.pp { color: #f5ce29; }
  code span.ss { color: #f3b2e0; }
  code span.va { color: #4d4add; }
}

/* The table of contents. Narrow: a button in the corner and a drawer, closed
   until the checkbox behind the button is checked. Wide: a fixed column on the
   left of the text, and no button. */
#toc-toggle { display: none; }
label[for="toc-toggle"] { position: fixed; right: 1rem; bottom: 1rem; z-index: 3; font: 600 14px system-ui, sans-serif; color: var(--bg); background: var(--accent); border-radius: 999px; padding: .6em 1.1em; cursor: pointer; box-shadow: 0 2px 10px rgba(0,0,0,.35); user-select: none; }
label[for="toc-toggle"] .toc-close, #toc-toggle:checked ~ label[for="toc-toggle"] .toc-open { display: none; }
#toc-toggle:checked ~ label[for="toc-toggle"] .toc-close { display: inline; }
nav#TOC { display: none; position: fixed; top: 0; left: 0; bottom: 0; z-index: 2; width: min(20rem, 85vw); overflow-y: auto; scrollbar-width: thin; box-sizing: border-box; padding: 1.2rem 1.4rem 5rem; background: var(--bg); border-right: 1px solid var(--rule); box-shadow: 0 0 30px rgba(0,0,0,.35); font: 15px/1.5 system-ui, sans-serif; }
#toc-toggle:checked ~ nav#TOC { display: block; }
nav#TOC h2 { font-size: 13px; text-transform: uppercase; letter-spacing: .08em; color: var(--muted); margin: 0 0 .4em; padding: 0; border: 0; }
nav#TOC ul { list-style: none; margin: 0; padding: 0; }
nav#TOC > ul > li:has(ul) { margin-top: .6em; font-weight: 600; }
nav#TOC ul ul { padding-left: 1rem; font-weight: 400; }
nav#TOC a { display: block; color: var(--fg); padding: .15em .5em; margin-left: -.5em; border-radius: 4px; }
nav#TOC a:hover { text-decoration: none; background: var(--code-bg); }
nav#TOC a.toc-current { color: var(--accent); background: color-mix(in srgb, var(--accent) 14%, transparent); }
@media (min-width: 76rem) {
  body { padding-left: 21rem; }
  label[for="toc-toggle"] { display: none; }
  nav#TOC { display: block; left: calc(50% - 35.5rem); width: 18rem; padding: 2.7rem 1rem 3rem 0; background: transparent; border: 0; box-shadow: none; }
}
</style>
<script type="module">
// A document with a single top-level heading doesn't need that heading listed
// in the contents; the sections under it are the contents.
const toc = document.getElementById('TOC'), h1 = document.querySelector('h1');
if (toc && h1) {
  const top = toc.querySelector(':scope > ul');
  if (top && top.children.length === 1 && top.querySelector(':scope > li > a')?.getAttribute('href') === '#' + h1.id) {
    const inner = top.querySelector(':scope > li > ul');
    if (inner) top.replaceWith(inner); else toc.remove();
  }
}
// As the page scrolls, the entry for the section in view is marked, and kept
// in view in the contents. Following a link closes the drawer. The history is
// not touched, so the back button undoes the link.
if (toc?.isConnected) {
  const links = new Map();
  for (const a of toc.querySelectorAll('a[href^="#"]')) links.set(a.getAttribute('href').slice(1), a);
  const heads = [...document.querySelectorAll('h1[id], h2[id]')].filter(h => links.has(h.id));
  let current;
  const spy = () => {
    let h = heads[0];
    for (const x of heads) { if (x.getBoundingClientRect().top <= innerHeight / 3) h = x; else break; }
    if (scrollY + innerHeight >= document.documentElement.scrollHeight - 2) h = heads.at(-1);
    if (!h || h === current) return;
    current = h;
    toc.querySelector('.toc-current')?.classList.remove('toc-current');
    const a = links.get(h.id);
    a.classList.add('toc-current');
    const r = a.getBoundingClientRect(), n = toc.getBoundingClientRect();
    if (r.top < n.top || r.bottom > n.bottom) toc.scrollTop += r.top - n.top - n.height / 2;
  };
  addEventListener('scroll', spy, { passive: true });
  addEventListener('resize', spy);
  // Scrolling that the browser does for itself, on the back button, doesn't
  // always announce itself; headings crossing the viewport always do.
  const seen = new IntersectionObserver(spy, { rootMargin: '0px 0px -60% 0px' });
  for (const h of heads) seen.observe(h);
  spy();
  const toggle = document.getElementById('toc-toggle');
  toc.addEventListener('click', e => { if (e.target.closest('a') && toggle) toggle.checked = false; });
}
// Chunk references are ordinary Go comments in the source; here they become
// links to the definition.
const slug = s => 'chunk-' + s.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
for (const span of document.querySelectorAll('pre code span.co, pre code span.cm')) {
  const m = span.textContent.match(/^\s*(?:\/\/|\/\*|--\[\[|--|#)\s*<<\s*(.+?)\s*>>\s*(?:\*\/|\]\])?\s*$/);
  if (!m) continue;
  const a = document.createElement('a');
  a.className = 'lit-ref'; a.href = '#' + slug(m[1].replace(/\s+/g, ' ')); a.textContent = '⟨' + m[1] + '⟩';
  span.replaceChildren(a);
}
const diagrams = document.querySelectorAll('pre.mermaid');
if (diagrams.length) {
  for (const pre of diagrams) { const code = pre.querySelector('code'); if (code) pre.textContent = code.textContent; }
  const { default: mermaid } = await import('https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs');
  mermaid.initialize({ startOnLoad: false, theme: matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'default' });
  await mermaid.run({ nodes: diagrams });
}
</script>
`

Typst

internal/weave/typst.go

For Typst, litgo’s additions are calls to functions the preamble defines, and they pass through pandoc as raw Typst. Text goes in as string literals, so nothing in a chunk name can be mistaken for markup.

func rawTypst(code string) string { return "```{=typst}\n" + code + "\n```\n" }

// typstString writes s as a Typst string literal.
func typstString(s string) string {
    return `"` + strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`, "\t", `\t`).Replace(s) + `"`
}

var typstTarget = target{
    chunk: func(id, name, sign string, uses []use) string {
        var pairs strings.Builder
        for _, u := range uses {
            pairs.WriteString("(" + typstString(u.ID) + ", " + typstString(u.Name) + "), ")
        }
        return rawTypst(fmt.Sprintf("#lit-chunk(%s, %s, %s, (%s))", typstString(id), typstString(name), typstString(sign), pairs.String()))
    },
    file: func(path string) string { return rawTypst("#lit-file(" + typstString(path) + ")") },
    literal: [2]string{
        rawTypst("#lit-literal.update(true)"),
        rawTypst("#lit-literal.update(false)"),
    },
    diagrams:  true,
    figure:    func(svg string) string { return rawTypst("#lit-figure(" + typstString(svg) + ")") },
    dropTitle: true,
}

The Typst source is a few definitions that describe the document, then the preamble, then pandoc’s rendering of the woven Markdown. Why the list of chunk anchors? Typst refuses to compile a link to a label that doesn’t exist, and one reference to an undefined chunk shouldn’t cost the reader the whole PDF.

//go:embed preamble.typ
var preamble string

// Typst returns the document as Typst markup.
func Typst(d *lit.Doc) (string, error) {
    t := typstTarget
    t.figures = drawn(d)
    body, err := pandoc(weave(d, t), "--to", "typst")
    if err != nil {
        return "", err
    }
    var anchors strings.Builder
    chapters := 0
    for _, c := range d.Chunks() {
        if len(c.Blocks) > 0 {
            anchors.WriteString(typstString(slug(c.Name)) + ", ")
        }
    }
    for i, l := range d.Lines {
        if strings.HasPrefix(l, "# ") && d.BlockAt(i) == nil {
            chapters++
        }
    }
    var sb strings.Builder
    fmt.Fprintf(&sb, "#let lit-title = %s\n", typstString(Title(d)))
    fmt.Fprintf(&sb, "#let lit-summary = %s\n", typstString(Summary(d)))
    fmt.Fprintf(&sb, "#let lit-chunks = (%s)\n", anchors.String())
    fmt.Fprintf(&sb, "#let lit-contents = %t\n", chapters > 1)
    sb.WriteString(preamble)
    sb.Write(body)
    return sb.String(), nil
}

// PDF weaves through Typst. The source goes in on standard input and the
// document's directory is the root, so images it refers to are found.
func PDF(d *lit.Doc, out string) error {
    src, err := Typst(d)
    if err != nil {
        return err
    }
    bin, err := exec.LookPath("typst")
    if err != nil {
        return fmt.Errorf("typst is not installed (https://typst.app); `litgo weave --to html` needs only pandoc")
    }
    cmd := exec.Command(bin, "compile", "--root", filepath.Dir(d.Path), "-", out)
    cmd.Dir = filepath.Dir(d.Path)
    cmd.Stdin = strings.NewReader(src)
    if msg, err := cmd.CombinedOutput(); err != nil {
        return fmt.Errorf("typst: %v\n%s\n(`litgo weave --to typ` writes the source for a closer look)", err, msg)
    }
    return nil
}

The preamble

internal/weave/preamble.typ

This sets up the page. Every font named here ships inside Typst, so the PDF looks the same on every machine. A document with several top-level headings is treated as a book: it gets a table of contents, and each chapter starts on a new page.

#set document(title: lit-title)
#set page(paper: "a4", margin: (x: 2.3cm, top: 2.4cm, bottom: 2.6cm), numbering: "1")
#set text(font: "Libertinus Serif", size: 10.5pt)
#set par(justify: true)
#show heading: set block(above: 1.8em, below: 0.9em)
#show heading.where(level: 1): it => {
  if it.outlined { pagebreak(weak: true) } // but not before "Contents"
  set text(size: 1.3em)
  it
}
#show link: set text(fill: rgb("#0a58ca"))
#set table(stroke: none, inset: (x: 6pt, y: 4pt))
#show figure.where(kind: table): set block(breakable: true)
#show figure.where(kind: table): set par(justify: false)

pandoc’s output expects these two names. Normally they come from pandoc’s own template.

#let divider() = line(length: 100%, stroke: 0.5pt + luma(170))
#let horizontalrule = divider()

Code sits on a tinted block. A drawn diagram arrives as a block in the language diagram, and gets set as a figure instead: centred, untinted, and never split across pages.

#show raw: set text(font: "DejaVu Sans Mono")
#show raw.where(block: false): set text(size: 0.92em)
#show raw.where(block: true): it => {
  set par(justify: false)
  if it.lang == "diagram" {
    // Lines set solid, so that box-drawing characters join up.
    set par(leading: 0pt)
    set text(size: 7.5pt, top-edge: "ascender", bottom-edge: "descender")
    align(center, block(breakable: false, above: 1.4em, below: 1.4em, it))
  } else {
    block(width: 100%, fill: luma(246), radius: 3pt, inset: (x: 8pt, y: 7pt), text(size: 7.8pt, it))
  }
}

A diagram that mermaid.js drew arrives as SVG. mermaid.js sizes it for a screen, with 16px text, so it gets scaled down to sit beside 10.5pt prose, and further still if that’s what it takes to fit the page.

#let lit-figure(svg) = layout(page => {
  let data = bytes(svg)
  let natural = measure(image(data, format: "svg"))
  let scale = calc.min(0.62, page.width / natural.width, 0.8 * page.height / natural.height)
  align(center, block(breakable: false, above: 1.4em, below: 1.4em,
    image(data, format: "svg", width: scale * natural.width)))
})

A reference in code is a comment holding a name in double angle brackets. The name is shown the way the editor shows it, and linked to its chunk if there is one. By the time a show rule sees the comment, the syntax highlighter has already chopped it into tokens, so the rule only looks for the brackets and leaves the comment marker alone. The rule is switched off between lit-literal.update(true) and (false).

#let lit-literal = state("lit-literal", false)
#let lit-reference = regex("<<\\s*([^<>]+?)\\s*>>")
#let lit-slug(name) = "chunk-" + lower(name).replace(regex("[^a-z0-9]+"), "-").trim("-")
#let lit-name(name) = text(font: "Libertinus Serif", style: "italic", size: 1.2em)[⟨#name⟩]
#let lit-link(id, name) = if id in lit-chunks { link(label(id), lit-name(name)) } else { lit-name(name) }

#show raw.where(block: true): it => {
  show lit-reference: found => context {
    if lit-literal.get() { found } else {
      let name = found.text.match(lit-reference).captures.at(0).replace(regex("\\s+"), " ")
      lit-link(lit-slug(name), name)
    }
  }
  it
}

This is the heading over a chunk: the chunk’s name and sign on the left, its uses on the right, and the label that references link to. It sticks to the block below it.

#let lit-chunk(id, name, sign, uses) = {
  let where = if uses.len() == 0 { [never used] } else {
    [used in ] + uses.map(u => if u.at(0) == "" { u.at(1) } else { lit-link(u.at(0), u.at(1)) }).join(", ")
  }
  let head = block(above: 1.5em, below: 0.5em, sticky: true, {
    lit-name(name) + h(0.4em) + text(size: 1.2em, sign) + h(1fr) + text(size: 8pt, fill: luma(110), where)
  })
  if id == "" { head } else [#head #label(id)]
}

#let lit-file(path) = block(above: 2.2em, below: 1em, sticky: true, {
  line(length: 100%, stroke: 0.5pt + luma(170))
  v(-0.4em)
  text(font: "DejaVu Sans Mono", size: 8.5pt, weight: "bold", fill: luma(90), "▸ " + path)
})

The title is lifted out of the body and set here, above the summary of the files the document tangles to.

#block(below: 0.8em, text(size: 22pt, weight: "bold", lit-title))
#block(below: 2em, text(size: 9pt, fill: luma(110), lit-summary))

#if lit-contents {
  outline(depth: 2, indent: auto)
  pagebreak(weak: true)
}
internal/weave/weave_test.go

pandoc and Typst might not be installed where the tests run, so the test stops short of them and checks what litgo would hand them.

const doc = "<!-- package: main -->\n\n# Title\n\n```mermaid\ngraph LR\n  a --> b\n```\n\n" +
    "```go\nfunc main() {\n\t// <<say \"hi\">>\n}\n```\n\n<!-- chunk: say \"hi\" -->\n```go\nprintln(1)\n```\n\n" +
    "<!-- notangle -->\n```go\n// <<only an illustration>>\n```\n"

func TestWeaveForTypst(t *testing.T) {
    d := lit.Parse("/x/t.lit.md", []byte(doc))
    out := weave(d, typstTarget)
    for _, want := range []string{
        `#lit-chunk("chunk-say-hi", "say \"hi\"", "≡", (("", "the program"), ))`,
        "```diagram\n┌───┐",
        "#lit-literal.update(true)\n```\n```go\n// <<only an illustration>>\n```\n```{=typst}\n#lit-literal.update(false)",
    } {
        if !strings.Contains(out, want) {
            t.Errorf("woven Markdown lacks %q:\n%s", want, out)
        }
    }
    if strings.Contains(out, "# Title") || strings.Contains(out, "```mermaid") {
        t.Errorf("the title and the diagram source should be gone:\n%s", out)
    }
    drew := typstTarget
    drew.figures = map[int]string{4: `<svg id="a"/>`}
    if out := weave(d, drew); !strings.Contains(out, `#lit-figure("<svg id=\"a\"/>")`) || strings.Contains(out, "```diagram") {
        t.Errorf("the drawn diagram should stand in for the block:\n%s", out)
    }
    if Title(d) != "Title" || Summary(d) != "package main · tangles to t.go" {
        t.Errorf("title %q, summary %q", Title(d), Summary(d))
    }
}

func TestWeaveForHTML(t *testing.T) {
    out := Markdown(lit.Parse("/x/t.lit.md", []byte(doc)))
    for _, want := range []string{
        "# Title\n\n<div class=\"lit-meta\">package main · tangles to t.go</div>",
        `<div class="lit-chunk" id="chunk-say-hi"><span class="lit-name">⟨say &#34;hi&#34;⟩ ≡</span> <span class="lit-uses">used in the program</span></div>`,
        "```mermaid",
    } {
        if !strings.Contains(out, want) {
            t.Errorf("woven Markdown lacks %q:\n%s", want, out)
        }
    }
}

Proofs

A test tells you the program did the right thing the times somebody looked. A proof tells you it does the right thing every time. For a long time only the first was practical, and the checking was never the hard part (a machine does that). The hard part was writing down what to check: the precondition, the invariant of every loop, the quantity that shrinks. A coding agent is good at proposing those, and a person is good at reading them. A literate program is their natural home, because the argument for why a loop is right is exactly what the prose around the loop was trying to say anyway.

VeGo (Verified Go, by Tina Massoudi and Chris Dutchyn) is a notation for that argument. The specifications are comments that start with //@, so an annotated program is still plain Go. It compiles, runs and formats exactly as before. litgo reads those comments and checks them.

//@ Requires n >= 0 ^ d > 0
func Divide(n, d int) (q, r int) {
    r = n
    //@ Invariant n = q*d + r ^ r >= 0
    //@ Variant r
    for r >= d {
        q, r = q+1, r-d
    }
    return q, r
}
//@ Ensures n = q*d + r ^ 0 <= r < d
flowchart LR
    doc[prog.lit.md] -->|tangle, in memory| go[annotated Go]
    go -->|walk every path| vcs[proof obligations]
    vcs -->|refute| prover[arithmetic prover]
    prover -->|not proved, at a line of the Markdown| doc

The paper’s own checker, vegop, hasn’t been published, so litgo brings its own, and it lives in this chapter. It follows the paper’s design where it counts. It runs natively in Go, with no SMT solver behind it (a general-purpose logic engine like Z3, which most verifiers lean on). It works through a function one assignment at a time, giving every assignment a new name the way SSA form does. And it settles its verification conditions with Fourier-Motzkin elimination over linear integer arithmetic.

It is a deductive verifier, and two things about it deserve to be said plainly.

What a proof covers: every Ensures on every path to every return; every Invariant, on entry and around the loop; termination, wherever a Variant or a Measure is given; every index inside its slice and every divisor not zero; the Requires of every call. What it doesn’t cover: integers are the mathematical ones, so overflow is out of sight (as it is in the paper), and a nil pointer to a struct is never considered.

A function without annotations is never looked at. You can verify a program one function at a time.

What there is to say

internal/vego/vego.go
Annotation Where Meaning
Requires P above a function The callers owe P; the body may assume it.
Ensures Q under the closing brace Holds at every return. A parameter means its value on entry, the results their final values.
Exsures X under the closing brace A state that no return can be in: ~X is proved.
Invariant I above a loop, or at the end of its body Holds on entry and after every iteration, and is all that is known after the loop, with the negated condition.
Variant e with the invariant An integer that is never negative while the loop runs and gets smaller every time round.
Measure e above a recursive function, or in it The same, for recursion: smaller at every recursive call.
BaseCase c in a body No recursive call is reachable once c has held.
InductionHypothesis P, Assert P, or just P in a body Proved there, and known from there on.
Preserves P in a body Proved there and again after every assignment until the block ends. A loop inside the block may assume it.
Axiom P in a body Assumed without proof, with a warning every time.
Predicate N(a, b) ::= P anywhere A name for a formula.

The paper’s .vgo dialect also has a while keyword, a skip statement and functions declared inside functions. None of those is Go, and a litgo block has to be Go, so they’re left out. for cond {}, an empty statement and a separate function say the same things anyway. Property and contracts on interface methods are read but not checked, and litgo says so.

// Marker starts a comment that is a VeGo annotation.
const Marker = "//@"

// Source is a Go file to verify.
type Source struct {
    Path string
    Text []byte
}

// Pos is a 0-based position in a Source.
type Pos struct {
    Path      string
    Line, Col int
}

// Finding is something the verifier has to say. Ref, when it has a Path, is a
// second position; the message mentions it as {ref}.
type Finding struct {
    Pos
    EndCol  int
    Warning bool
    Message string
    Ref     Pos
}

// Summary counts what a verification did.
type Summary struct {
    Functions   int // functions with annotations
    Proved      int // of which every obligation was proved
    Obligations int
    Names       []string // the proved functions
}

An annotation is one //@ comment. A formula too long for one line carries on in the next comment, under the same rule Go uses for semicolons: a line that ends in something that can’t end a formula (an operator, an opening bracket) isn’t finished.

// annot is one annotation: a //@ comment, with the lines that continue it.
type annot struct {
    kind    string
    text    string // the formula
    e       *expr
    pos     token.Pos
    where   Pos
    endCol  int
    textCol int // where the formula starts on the line
    first   int // how much of the formula is on that line
    lines   int // how many comment lines it takes
    used    bool
}

// Annotated reports whether a Go file has annotations at all, which is what
// decides whether verifying it is worth the time.
func Annotated(text []byte) bool {
    return strings.Contains(string(text), Marker) || strings.Contains(string(text), "// @ ")
}

Prove is the front door, where the rest of litgo comes in. It takes what a document tangles to, in memory, and answers in the document’s own lines, which is the source map’s job as always. Even a position mentioned inside a message (the return a postcondition failed for, say) gets translated. A document’s Go files are verified one directory at a time, because a contract in one file is used by calls in another.

// Prove verifies the annotated Go a document tangles to, and reports at the
// document's own lines. Nothing is written, so it works on a buffer.
func Prove(res *lit.Result) ([]lit.Diag, Summary) {
    var total Summary
    if res.HasErrors() {
        return nil, total
    }
    packages := map[string][]*lit.Output{}
    var dirs []string
    for _, o := range res.Files {
        if o.File.Lang != "go" || strings.HasSuffix(o.Path, "_test.go") {
            continue
        }
        dir := filepath.Dir(o.Path)
        if packages[dir] == nil {
            dirs = append(dirs, dir)
        }
        packages[dir] = append(packages[dir], o)
    }
    var diags []lit.Diag
    seen := map[string]bool{}
    for _, dir := range dirs {
        var sources []Source
        outputs := map[string]*lit.Output{}
        annotated := false
        for _, o := range packages[dir] {
            text := o.Bytes()
            annotated = annotated || Annotated(text)
            sources = append(sources, Source{o.Path, text})
            outputs[o.Path] = o
        }
        if !annotated {
            continue
        }
        findings, sum := Verify(sources)
        total.Functions += sum.Functions
        total.Proved += sum.Proved
        total.Obligations += sum.Obligations
        total.Names = append(total.Names, sum.Names...)
        for _, f := range findings {
            o := outputs[f.Path]
            line, col, _, ok := o.Map(f.Line, f.Col)
            if !ok {
                continue
            }
            d := lit.Diag{File: res.Doc.Path, Line: line, Col: col, Severity: lit.SevError, Message: f.Message, Source: "vego"}
            if f.Warning {
                d.Severity = lit.SevWarning
            }
            if f.EndCol > f.Col {
                d.EndCol = col + f.EndCol - f.Col
            }
            if f.Ref.Path != "" {
                if l, _, _, ok := outputs[f.Ref.Path].Map(f.Ref.Line, f.Ref.Col); ok {
                    d.Message = strings.ReplaceAll(d.Message, "{ref}", fmt.Sprintf("line %d", l+1))
                }
            }
            // A chunk that is used twice is verified twice, and fails twice.
            key := fmt.Sprintf("%d:%d:%s", d.Line, d.Col, d.Message)
            if !seen[key] {
                seen[key] = true
                diags = append(diags, d)
            }
        }
    }
    return diags, total
}

Verify is the same thing without the Markdown, for one package of Go. A function counts as proved when verifying it reported no error. Warnings (an Axiom, a loop without a Variant) don’t take a proof away. They qualify it.

// Verify checks every annotated function of one package.
func Verify(sources []Source) ([]Finding, Summary) {
    v := &verifier{fset: token.NewFileSet(), funcs: map[string]*fn{}, structs: map[string]map[string]string{},
        consts: map[string]int64{}, preds: map[string]*predicate{}, reported: map[string]bool{}}
    for _, s := range sources {
        f, err := parser.ParseFile(v.fset, s.Path, s.Text, parser.ParseComments|parser.SkipObjectResolution)
        if err != nil {
            continue // the syntax check has more to say about it than we do
        }
        v.files = append(v.files, f)
    }
    for _, f := range v.files {
        v.declarations(f)
    }
    for _, f := range v.files {
        v.functions(f)
    }
    for _, f := range v.files {
        v.annotations(f)
    }
    var sum Summary
    for _, f := range v.ordered {
        if !f.annotated {
            continue
        }
        sum.Functions++
        before := v.errors
        v.verify(f)
        sum.Obligations += f.obligations
        if v.errors == before && !f.broken {
            sum.Proved++
            sum.Names = append(sum.Names, f.key)
        }
    }
    for _, a := range v.all {
        if !a.used && a.e != nil {
            v.report(a.where, a.endCol, false, a.kind+" is not attached to anything: Requires goes above a function, Ensures below it, Invariant and Variant at a loop")
        }
    }
    sort.SliceStable(v.findings, func(i, j int) bool {
        a, b := v.findings[i], v.findings[j]
        if a.Path != b.Path {
            return a.Path < b.Path
        }
        return a.Line < b.Line
    })
    return v.findings, sum
}

The verifier is purely syntactic. It never runs the type checker, because the types it can reason about are few enough to read straight off the declarations: the integer types, bool, []int, and the package’s own structs, of which it uses the integer and boolean fields.

type vkind int

const (
    kNone vkind = iota
    kInt
    kBool
    kSlice
    kStruct
)

type param struct {
    name string
    kind vkind
    typ  string // the struct's name, for kStruct
}

// fn is a function of the package, with its contract.
type fn struct {
    key         string // name, or Type.name for a method
    decl        *ast.FuncDecl
    recv        *param
    params      []param
    results     []param
    requires    []*annot
    ensures     []*annot
    exsures     []*annot
    measure     []*annot
    inside      []*annot // the annotations in the body
    annotated   bool
    broken      bool // an annotation of it does not parse
    obligations int
}

type verifier struct {
    fset       *token.FileSet
    files      []*ast.File
    funcs      map[string]*fn
    ordered    []*fn
    structs    map[string]map[string]string
    consts     map[string]int64
    preds      map[string]*predicate
    all        []*annot
    findings   []Finding
    reported   map[string]bool
    errors     int
    fresh      int
    parseFresh int
}
func (v *verifier) position(p token.Pos) Pos {
    pp := v.fset.Position(p)
    return Pos{pp.Filename, pp.Line - 1, pp.Column - 1}
}

func (v *verifier) report(at Pos, endCol int, warning bool, msg string) {
    v.reportRef(at, endCol, warning, msg, Pos{})
}

// reportRef reports once per place and message: a contract that fails on
// three paths is still one problem.
func (v *verifier) reportRef(at Pos, endCol int, warning bool, msg string, ref Pos) {
    key := fmt.Sprintf("%s:%d:%d:%s", at.Path, at.Line, at.Col, msg)
    if v.reported[key] {
        return
    }
    v.reported[key] = true
    if !warning {
        v.errors++
    }
    v.findings = append(v.findings, Finding{Pos: at, EndCol: endCol, Warning: warning, Message: msg, Ref: ref})
}
// kindOf reads a type the way the verifier sees it: integers, booleans,
// slices of integers, and structs declared in the package.
func (v *verifier) kindOf(t ast.Expr) (vkind, string) {
    switch t := t.(type) {
    case *ast.Ident:
        switch t.Name {
        case "int", "int8", "int16", "int32", "int64":
            return kInt, ""
        case "bool":
            return kBool, ""
        }
        if _, ok := v.structs[t.Name]; ok {
            return kStruct, t.Name
        }
    case *ast.StarExpr:
        if k, name := v.kindOf(t.X); k == kStruct {
            return k, name
        }
    case *ast.ArrayType:
        if k, _ := v.kindOf(t.Elt); t.Len == nil && k == kInt {
            return kSlice, ""
        }
    case *ast.ParenExpr:
        return v.kindOf(t.X)
    }
    return kNone, ""
}

// declarations notes the structs and the integer constants.
func (v *verifier) declarations(f *ast.File) {
    for _, d := range f.Decls {
        g, ok := d.(*ast.GenDecl)
        if !ok {
            continue
        }
        for _, spec := range g.Specs {
            switch s := spec.(type) {
            case *ast.TypeSpec:
                if st, ok := s.Type.(*ast.StructType); ok {
                    fields := map[string]string{}
                    for _, fl := range st.Fields.List {
                        for _, n := range fl.Names {
                            fields[n.Name] = typeText(fl.Type)
                        }
                    }
                    v.structs[s.Name.Name] = fields
                }
            case *ast.ValueSpec:
                if g.Tok != token.CONST {
                    continue
                }
                for i, n := range s.Names {
                    if i < len(s.Values) {
                        if c, ok := constInt(s.Values[i]); ok {
                            v.consts[n.Name] = c
                        }
                    }
                }
            }
        }
    }
}

// functions notes the functions, once every struct they may mention is known.
func (v *verifier) functions(f *ast.File) {
    for _, d := range f.Decls {
        fd, ok := d.(*ast.FuncDecl)
        if !ok || fd.Body == nil {
            continue
        }
        f := &fn{key: fd.Name.Name, decl: fd}
        if fd.Recv != nil && len(fd.Recv.List) == 1 {
            r := fd.Recv.List[0]
            k, typ := v.kindOf(r.Type)
            name := "_"
            if len(r.Names) == 1 {
                name = r.Names[0].Name
            }
            f.recv = &param{name, k, typ}
            f.key = strings.TrimPrefix(typeText(r.Type), "*") + "." + f.key
        }
        f.params = v.fields(fd.Type.Params, "")
        f.results = v.fields(fd.Type.Results, "ret_")
        v.funcs[f.key] = f
        v.ordered = append(v.ordered, f)
    }
}

func (v *verifier) fields(list *ast.FieldList, unnamed string) []param {
    var out []param
    if list == nil {
        return nil
    }
    for _, fl := range list.List {
        k, typ := v.kindOf(fl.Type)
        if len(fl.Names) == 0 {
            out = append(out, param{fmt.Sprintf("%s%d", unnamed, len(out)), k, typ})
        }
        for _, n := range fl.Names {
            out = append(out, param{n.Name, k, typ})
        }
    }
    return out
}

func typeText(t ast.Expr) string {
    switch t := t.(type) {
    case *ast.Ident:
        return t.Name
    case *ast.StarExpr:
        return "*" + typeText(t.X)
    case *ast.ArrayType:
        if t.Len == nil {
            return "[]" + typeText(t.Elt)
        }
    }
    return "?"
}

func constInt(e ast.Expr) (int64, bool) {
    switch e := e.(type) {
    case *ast.BasicLit:
        if e.Kind == token.INT {
            var v int64
            if _, err := fmt.Sscan(e.Value, &v); err == nil {
                return v, true
            }
        }
    case *ast.UnaryExpr:
        if c, ok := constInt(e.X); ok && e.Op == token.SUB {
            return -c, true
        }
    case *ast.ParenExpr:
        return constInt(e.X)
    }
    return 0, false
}

Where an annotation stands decides whose it is. Requires belongs to the function below it, and Ensures to the function above it, which is where VeGo puts it: a postcondition reads best after the code, the way a conclusion does. An Ensures in a function’s doc comment, next to its Requires, is accepted too. Annotations inside a body wait there until the walk through the body reaches them.

gofmt has opinions about comments, and an annotation has to survive them, because code gets formatted on save whether anyone is thinking about annotations or not. gofmt does two things to them. In a doc comment it puts a space after the slashes, so //@ Requires above a function comes back as // @ Requires. And it puts a blank line between a closing brace and a comment under it, so an Ensures drifts loose from its function. Neither one changes the meaning here: // @ followed by a keyword reads as //@, and an Ensures looks upward past blank lines. (lit.ProofComment, which decides what counts as an annotation, lives in Finding the annotations, because the editor’s highlighting asks the same question.)

// annotations reads a file's //@ comments, parses them, and hands each to the
// function it belongs to.
func (v *verifier) annotations(f *ast.File) {
    var list []*annot
    for _, group := range f.Comments {
        var open *annot // an annotation whose formula is not finished
        for _, c := range group.List {
            body, ok := lit.ProofComment(c.Text, open != nil)
            if !ok {
                open = nil
                continue
            }
            if open != nil {
                open.text += " " + strings.TrimSpace(body)
                open.lines++
                if !continues(body) {
                    open = nil
                }
                continue
            }
            rest := strings.TrimLeft(body, " \t")
            a := &annot{kind: "Assert", pos: c.Pos(), where: v.position(c.Pos()), lines: 1}
            a.endCol = a.where.Col + len(c.Text)
            word, after, _ := strings.Cut(rest, " ")
            if lit.ProofKeywords[word] {
                a.kind = word
                rest = strings.TrimLeft(after, " \t")
            }
            a.text = strings.TrimSpace(rest)
            a.first = len(a.text)
            a.textCol = a.where.Col + len(c.Text) - len(rest)
            list = append(list, a)
            if continues(body) {
                open = a
            }
        }
    }
    for _, a := range list {
        v.parse(a)
    }
    v.all = append(v.all, list...)

    var decls []*fn
    for _, fd := range v.ordered {
        if v.fset.File(fd.decl.Pos()) == v.fset.File(f.Pos()) {
            decls = append(decls, fd)
        }
    }
    for _, a := range list {
        if a.kind == "Predicate" {
            a.used = true
            continue
        }
        if v.inType(f, a) {
            v.report(a.where, a.endCol, true, "litgo prove does not check contracts on interface methods")
            a.used = true
            continue
        }
        var inside, before, after *fn
        for _, fd := range decls {
            switch {
            case a.pos > fd.decl.Body.Lbrace && a.pos < fd.decl.Body.Rbrace:
                inside = fd
            case fd.decl.End() <= a.pos:
                before = fd
            case a.pos < fd.decl.Pos() && after == nil:
                after = fd
            }
        }
        // An Ensures belongs to the function above it, unless it is part of
        // the doc comment of the function below and does not touch the one above.
        if after != nil && after.decl.Doc != nil && a.pos >= after.decl.Doc.Pos() && a.pos < after.decl.Doc.End() &&
            !(before != nil && v.adjacent(before, a, list)) {
            before = nil
        }
        var owner *fn
        switch a.kind {
        case "Requires":
            if owner = after; inside != nil {
                owner = nil
            }
            if owner != nil {
                owner.requires = append(owner.requires, a)
            }
        case "Ensures", "Exsures":
            if inside == nil {
                if owner = before; owner == nil {
                    owner = after
                }
            }
            if owner != nil && a.kind == "Ensures" {
                owner.ensures = append(owner.ensures, a)
            } else if owner != nil {
                owner.exsures = append(owner.exsures, a)
            }
        case "Measure":
            if owner = inside; owner == nil {
                owner = after
            }
            if owner != nil {
                owner.measure = append(owner.measure, a)
            }
        case "Property":
            v.report(a.where, a.endCol, true, "litgo prove does not check Property annotations")
            a.used = true
            continue
        default:
            if owner = inside; owner != nil {
                owner.inside = append(owner.inside, a)
            }
        }
        if owner != nil {
            owner.annotated = true
            owner.broken = owner.broken || a.e == nil
            if a.kind != "Requires" && a.kind != "Ensures" && a.kind != "Exsures" && a.kind != "Measure" {
                continue // the body marks its own as used, when it reaches them
            }
            a.used = true
        }
    }
}
// inType reports whether an annotation is written inside a type declaration,
// which is where VeGo puts the contract of an interface method.
func (v *verifier) inType(f *ast.File, a *annot) bool {
    for _, d := range f.Decls {
        if g, ok := d.(*ast.GenDecl); ok && g.Tok == token.TYPE && a.pos > g.Pos() && a.pos < g.End() {
            return true
        }
    }
    return false
}

// adjacent reports whether nothing but annotations lies between the end of a
// function and an annotation below it.
func (v *verifier) adjacent(f *fn, a *annot, list []*annot) bool {
    line := v.position(f.decl.End()).Line
    for _, b := range list {
        if b.where.Line == line+1 && b.pos > f.decl.End() {
            line += b.lines
            if b == a {
                return true
            }
        }
    }
    return false
}

func (v *verifier) parse(a *annot) {
    var err error
    switch a.kind {
    case "Predicate":
        var p *predicate
        if p, err = parsePredicate(a.text, &v.parseFresh); err == nil {
            v.preds[p.name] = p
            a.e = p.body
        }
    case "Immutable":
        a.e = eTrue
    default:
        a.e, err = parseFormula(a.text, &v.parseFresh)
        number := a.kind == "Variant" || a.kind == "Measure"
        if err == nil && a.e.op != opCall && isBool(a.e) == number {
            a.e = nil
            err = &syntaxError{0, "this has to be a formula, something that is true or false"}
            if number {
                err = &syntaxError{0, "this has to be an integer expression, something that can get smaller"}
            }
        }
    }
    if se, ok := err.(*syntaxError); ok {
        at := a.where
        at.Col = a.textCol + min(se.pos, a.first)
        v.report(at, a.endCol, false, "in this "+a.kind+": "+se.msg)
    }
}

The language of formulas

internal/vego/logic.go

Formulas are written the way you’d write them on a blackboard, in ASCII:

^ v ~ -> <- <-> and, or, not, implies, is implied by, if and only if
= <> < <= > >= comparisons, which chain: 0 <= i < n (== and != work too)
+ - * / % integer arithmetic, with Go’s division
len(A), A[i] the length and the elements of a slice
Forall k in [a, b) : P, Exists, Unique quantifiers over an interval, with [ ] for an end that is included and ( ) for one that is not, ... for no end at all, or over all of Z
A[a:b) <= m, m in A[a:b) every element of the range is at most m; some element is m
x' x after the assignment, where x is before it

The length of a slice can also be written |A|, and Go’s &&, || and ! are read as ^, v and ~, for the fingers that type them anyway.

One type serves for terms, for formulas, and later for the symbolic values of program variables. That way a Go expression and an annotation can meet in the same formula.

type op int

const (
    opNum op = iota
    opVar
    opTrue
    opFalse
    opAdd
    opSub
    opMul
    opDiv
    opMod
    opNeg
    opSel   // args: array, index
    opLen   // args: array
    opCall  // name(args): a predicate, or a Go function that is one expression
    opRange // name[lo:hi) as one side of a comparison; args: lo, hi (nil when omitted)
    opLt
    opLe
    opEq
    opNe
    opIn // x in A[lo:hi)
    opAnd
    opOr
    opNot
    opImp
    opIff
    opForall // name is the bound variable, args[0] the body
    opExists
)

type sort_ int

const (
    sInt sort_ = iota
    sBool
    sArr
)

// expr is a term or a formula. Annotations parse to it, Go expressions are
// translated to it, and the symbolic values of program variables are made of it.
type expr struct {
    op     op
    val    int64
    name   string
    primes int
    sort   sort_
    args   []*expr
    pos    int     // byte offset in the annotation's text
    open   [2]bool // opRange: is the low, the high end exclusive
}
func num(v int64) *expr              { return &expr{op: opNum, val: v} }
func sym(name string, s sort_) *expr { return &expr{op: opVar, name: name, sort: s} }
func mk(o op, args ...*expr) *expr   { return &expr{op: o, args: args} }

var (
    eTrue  = &expr{op: opTrue}
    eFalse = &expr{op: opFalse}
)

func and(a, b *expr) *expr {
    if a.op == opTrue {
        return b
    }
    if b.op == opTrue {
        return a
    }
    return mk(opAnd, a, b)
}

func not(a *expr) *expr { return mk(opNot, a) }

func conj(list []*expr) *expr {
    out := eTrue
    for _, e := range list {
        out = and(out, e)
    }
    return out
}

// isBool says whether e is a formula rather than an integer term.
func isBool(e *expr) bool {
    switch e.op {
    case opTrue, opFalse, opLt, opLe, opEq, opNe, opIn, opAnd, opOr, opNot, opImp, opIff, opForall, opExists:
        return true
    case opVar:
        return e.sort == sBool
    }
    return false
}
func (e *expr) String() string {
    bin := map[op]string{opAdd: "+", opSub: "-", opMul: "*", opDiv: "/", opMod: "%", opLt: "<", opLe: "<=",
        opEq: "=", opNe: "<>", opAnd: "^", opOr: "v", opImp: "->", opIff: "<->", opIn: "in"}
    switch e.op {
    case opNum:
        return strconv.FormatInt(e.val, 10)
    case opVar:
        return e.name + strings.Repeat("'", e.primes)
    case opTrue:
        return "true"
    case opFalse:
        return "false"
    case opNeg:
        return "-" + e.args[0].String()
    case opNot:
        return "~" + e.args[0].String()
    case opSel:
        return e.args[0].String() + "[" + e.args[1].String() + "]"
    case opLen:
        return "|" + e.args[0].String() + "|"
    case opCall:
        parts := make([]string, len(e.args))
        for i, a := range e.args {
            parts[i] = a.String()
        }
        return e.name + "(" + strings.Join(parts, ", ") + ")"
    case opRange:
        return e.name + "[:]"
    case opForall, opExists:
        q := "Forall "
        if e.op == opExists {
            q = "Exists "
        }
        return "(" + q + e.name + " : " + e.args[0].String() + ")"
    }
    return "(" + e.args[0].String() + " " + bin[e.op] + " " + e.args[1].String() + ")"
}

Everything done to a formula (giving names their values, unfolding a predicate, instantiating a quantifier) is a substitution. And there’s one thing a substitution must never do: touch a variable that a quantifier binds.

// rewrite rebuilds e bottom-up. f sees every variable that is not bound by a
// quantifier inside e, and returns its replacement or nil to keep it.
func rewrite(e *expr, f func(v *expr) *expr) *expr {
    return rewriteBound(e, f, nil)
}

func rewriteBound(e *expr, f func(v *expr) *expr, bound []string) *expr {
    if e == nil {
        return nil
    }
    switch e.op {
    case opNum, opTrue, opFalse:
        return e
    case opVar:
        for _, b := range bound {
            if b == e.name && e.primes == 0 {
                return e
            }
        }
        if r := f(e); r != nil {
            return r
        }
        return e
    }
    c := *e
    c.args = make([]*expr, len(e.args))
    if e.op == opForall || e.op == opExists {
        bound = append(bound[:len(bound):len(bound)], e.name)
    }
    for i, a := range e.args {
        c.args[i] = rewriteBound(a, f, bound)
    }
    return &c
}

// subst replaces the free variable name by r.
func subst(e *expr, name string, r *expr) *expr {
    return rewrite(e, func(v *expr) *expr {
        if v.name == name && v.primes == 0 {
            return r
        }
        return nil
    })
}

// A predicate is a named formula with parameters; using it means its body.
type predicate struct {
    name   string
    params []string
    body   *expr
}

Reading a formula

The letter v means or when it stands where an operator can stand, and it’s a variable anywhere else, so a program can still have a variable called v. A name may contain dots, which is how s.Len is one name, and it may end in primes.

type lexeme struct {
    kind string // "num", "id", "op", "eof"
    text string
    pos  int
}

type syntaxError struct {
    pos int
    msg string
}

func (e *syntaxError) Error() string { return e.msg }

var operators = []string{"<->", "::=", "...", "->", "<-", "<=", ">=", "<>", "==", "!=", "&&", "||",
    "^", "~", "!", "=", "<", ">", "+", "-", "*", "/", "%", "(", ")", "[", "]", ",", ":", "|"}

func lex(s string) ([]lexeme, error) {
    var out []lexeme
    for i := 0; i < len(s); {
        c := s[i]
        switch {
        case c == ' ' || c == '\t':
            i++
        case c >= '0' && c <= '9':
            j := i
            for j < len(s) && s[j] >= '0' && s[j] <= '9' {
                j++
            }
            out = append(out, lexeme{"num", s[i:j], i})
            i = j
        case c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z':
            j := i
            for j < len(s) && (s[j] == '_' || s[j] == '.' && j+1 < len(s) && s[j+1] != '.' || s[j] >= 'a' && s[j] <= 'z' || s[j] >= 'A' && s[j] <= 'Z' || s[j] >= '0' && s[j] <= '9') {
                j++
            }
            for j < len(s) && s[j] == '\'' {
                j++
            }
            out = append(out, lexeme{"id", s[i:j], i})
            i = j
        default:
            found := false
            for _, o := range operators {
                if strings.HasPrefix(s[i:], o) {
                    out = append(out, lexeme{"op", o, i})
                    i += len(o)
                    found = true
                    break
                }
            }
            if !found {
                return nil, &syntaxError{i, fmt.Sprintf("unexpected %q", string(c))}
            }
        }
    }
    return append(out, lexeme{"eof", "", len(s)}), nil
}
// continues reports whether an annotation line ends in a token that cannot
// end a formula, which is how a formula says it goes on on the next line.
func continues(text string) bool {
    toks, err := lex(text)
    if err != nil || len(toks) < 2 {
        return false
    }
    last := toks[len(toks)-2]
    if last.kind == "id" {
        return last.text == "v" || last.text == "in"
    }
    if last.kind != "op" {
        return false
    }
    switch last.text {
    case ")", "]", "...", "|":
        return false
    }
    return true
}

The parser is recursive descent, one function per level of binding strength, loosest first: <->, then ->, v, ^, ~, the comparisons, and the arithmetic. A syntax error is a panic that carries its column and gets caught at the top, so the error lands right under the character that caused it.

type reader struct {
    toks  []lexeme
    i     int
    fresh *int
}

func (p *reader) peek() lexeme { return p.toks[p.i] }
func (p *reader) next() lexeme {
    t := p.toks[p.i]
    if t.kind != "eof" {
        p.i++
    }
    return t
}

func (p *reader) is(text string) bool {
    t := p.peek()
    return t.kind == "op" && t.text == text
}

func (p *reader) isWord(text string) bool {
    t := p.peek()
    return t.kind == "id" && t.text == text
}

func (p *reader) fail(t lexeme, format string, args ...any) {
    panic(&syntaxError{t.pos, fmt.Sprintf(format, args...)})
}

func (p *reader) expect(text string) {
    if t := p.next(); t.kind != "op" || t.text != text {
        what := t.text
        if t.kind == "eof" {
            what = "the end of the annotation"
        }
        p.fail(t, "expected %q, found %s", text, what)
    }
}

// parseFormula parses a whole annotation body.
func parseFormula(text string, fresh *int) (e *expr, err error) {
    toks, err := lex(text)
    if err != nil {
        return nil, err
    }
    p := &reader{toks: toks, fresh: fresh}
    defer func() {
        if r := recover(); r != nil {
            se, ok := r.(*syntaxError)
            if !ok {
                panic(r)
            }
            e, err = nil, se
        }
    }()
    e = p.iff()
    if t := p.peek(); t.kind != "eof" {
        p.fail(t, "unexpected %q", t.text)
    }
    return e, nil
}

// parsePredicate parses "Name(a, b) ::= body".
func parsePredicate(text string, fresh *int) (pr *predicate, err error) {
    toks, err := lex(text)
    if err != nil {
        return nil, err
    }
    p := &reader{toks: toks, fresh: fresh}
    defer func() {
        if r := recover(); r != nil {
            se, ok := r.(*syntaxError)
            if !ok {
                panic(r)
            }
            pr, err = nil, se
        }
    }()
    name := p.next()
    if name.kind != "id" {
        p.fail(name, "a predicate needs a name")
    }
    pr = &predicate{name: name.text}
    p.expect("(")
    for !p.is(")") {
        t := p.next()
        if t.kind != "id" {
            p.fail(t, "expected a parameter name")
        }
        pr.params = append(pr.params, t.text)
        if !p.is(")") {
            p.expect(",")
        }
    }
    p.expect(")")
    p.expect("::=")
    pr.body = p.iff()
    if t := p.peek(); t.kind != "eof" {
        p.fail(t, "unexpected %q", t.text)
    }
    return pr, nil
}
func (p *reader) iff() *expr {
    l := p.imp()
    for p.is("<->") {
        t := p.next()
        l = &expr{op: opIff, args: []*expr{l, p.imp()}, pos: t.pos}
    }
    return l
}

func (p *reader) imp() *expr {
    l := p.or()
    if p.is("->") {
        t := p.next()
        return &expr{op: opImp, args: []*expr{l, p.imp()}, pos: t.pos}
    }
    if p.is("<-") {
        t := p.next()
        return &expr{op: opImp, args: []*expr{p.imp(), l}, pos: t.pos}
    }
    return l
}

func (p *reader) or() *expr {
    l := p.and()
    for p.isWord("v") || p.is("||") {
        t := p.next()
        l = &expr{op: opOr, args: []*expr{l, p.and()}, pos: t.pos}
    }
    return l
}

func (p *reader) and() *expr {
    l := p.neg()
    for p.is("^") || p.is("&&") {
        t := p.next()
        l = &expr{op: opAnd, args: []*expr{l, p.neg()}, pos: t.pos}
    }
    return l
}

func (p *reader) neg() *expr {
    if p.is("~") || p.is("!") {
        t := p.next()
        return &expr{op: opNot, args: []*expr{p.neg()}, pos: t.pos}
    }
    if p.isWord("Forall") || p.isWord("Exists") || p.isWord("Unique") {
        return p.quantifier()
    }
    return p.comparison()
}

A chain of comparisons is the conjunction of its links. A comparison with a range of a slice on one side is a quantifier in disguise, and the parser pulls off the mask: A[0:i) <= m becomes Forall k : 0 <= k < i -> A[k] <= m, and m in A[0:i) becomes an Exists.

var comparisons = map[string]bool{"=": true, "==": true, "<>": true, "!=": true, "<": true, "<=": true, ">": true, ">=": true}

// comparison parses a chain: 0 <= i < n means 0 <= i and i < n.
func (p *reader) comparison() *expr {
    l := p.sum()
    var out *expr
    for {
        t := p.peek()
        in := t.kind == "id" && t.text == "in"
        if !in && !(t.kind == "op" && comparisons[t.text]) {
            break
        }
        p.next()
        r := p.sum()
        c := p.compare(t, l, r)
        if out == nil {
            out = c
        } else {
            out = &expr{op: opAnd, args: []*expr{out, c}, pos: t.pos}
        }
        l = r
    }
    if out == nil {
        if l.op == opRange {
            p.fail(p.peek(), "a range like %s[a:b) can only be compared with something", l.name)
        }
        return l
    }
    return out
}

func (p *reader) compare(t lexeme, l, r *expr) *expr {
    if l.op == opRange || r.op == opRange {
        return p.overRange(t, l, r)
    }
    switch t.text {
    case "=", "==":
        if isBool(l) || isBool(r) {
            return &expr{op: opIff, args: []*expr{l, r}, pos: t.pos}
        }
        return &expr{op: opEq, args: []*expr{l, r}, pos: t.pos}
    case "<>", "!=":
        if isBool(l) || isBool(r) {
            return &expr{op: opNot, args: []*expr{{op: opIff, args: []*expr{l, r}, pos: t.pos}}, pos: t.pos}
        }
        return &expr{op: opNe, args: []*expr{l, r}, pos: t.pos}
    case "<":
        return &expr{op: opLt, args: []*expr{l, r}, pos: t.pos}
    case "<=":
        return &expr{op: opLe, args: []*expr{l, r}, pos: t.pos}
    case ">":
        return &expr{op: opLt, args: []*expr{r, l}, pos: t.pos}
    case ">=":
        return &expr{op: opLe, args: []*expr{r, l}, pos: t.pos}
    }
    p.fail(t, "%q needs a range like A[a:b) on its right", t.text)
    return nil
}

// overRange turns a comparison with A[lo:hi) into a quantifier over a fresh
// index: an order comparison holds for all elements, = and in for some.
func (p *reader) overRange(t lexeme, l, r *expr) *expr {
    if l.op == opRange && r.op == opRange {
        p.fail(t, "only one side of a comparison can be a range")
    }
    rng := r
    if l.op == opRange {
        rng = l
    }
    *p.fresh++
    idx := &expr{op: opVar, name: fmt.Sprintf("_idx%d", *p.fresh)}
    elem := &expr{op: opSel, args: []*expr{{op: opVar, name: rng.name, sort: sArr}, idx}}
    lo, hi := rng.args[0], rng.args[1]
    if lo == nil {
        lo = num(0)
    }
    if hi == nil {
        hi = &expr{op: opLen, args: []*expr{{op: opVar, name: rng.name, sort: sArr}}}
    }
    within := mk(opAnd, mk(bound(rng.open[0]), lo, idx), mk(bound(rng.open[1]), idx, hi))
    if rng.args[1] == nil {
        within.args[1] = mk(opLt, idx, hi)
    }
    if t.text == "in" || t.text == "=" || t.text == "==" {
        other := l
        if l.op == opRange {
            other = r
        }
        return &expr{op: opExists, name: idx.name, args: []*expr{mk(opAnd, within, mk(opEq, other, elem))}, pos: t.pos}
    }
    if t.text == "<>" || t.text == "!=" {
        other := l
        if l.op == opRange {
            other = r
        }
        return &expr{op: opForall, name: idx.name, args: []*expr{mk(opImp, within, mk(opNe, other, elem))}, pos: t.pos}
    }
    var c *expr
    if l.op == opRange {
        c = p.compare(t, elem, r)
    } else {
        c = p.compare(t, l, elem)
    }
    return &expr{op: opForall, name: idx.name, args: []*expr{mk(opImp, within, c)}, pos: t.pos}
}

func bound(open bool) op {
    if open {
        return opLt
    }
    return opLe
}
func (p *reader) sum() *expr {
    l := p.product()
    for p.is("+") || p.is("-") {
        t := p.next()
        o := opAdd
        if t.text == "-" {
            o = opSub
        }
        l = &expr{op: o, args: []*expr{l, p.product()}, pos: t.pos}
    }
    return l
}

func (p *reader) product() *expr {
    l := p.unary()
    for p.is("*") || p.is("/") || p.is("%") {
        t := p.next()
        o := map[string]op{"*": opMul, "/": opDiv, "%": opMod}[t.text]
        l = &expr{op: o, args: []*expr{l, p.unary()}, pos: t.pos}
    }
    return l
}

func (p *reader) unary() *expr {
    if p.is("-") {
        t := p.next()
        return &expr{op: opNeg, args: []*expr{p.unary()}, pos: t.pos}
    }
    return p.primary()
}

func (p *reader) primary() *expr {
    t := p.next()
    switch {
    case t.kind == "num":
        v, err := strconv.ParseInt(t.text, 10, 64)
        if err != nil {
            p.fail(t, "%s is too large", t.text)
        }
        return &expr{op: opNum, val: v, pos: t.pos}
    case t.kind == "op" && t.text == "(":
        e := p.iff()
        p.expect(")")
        return e
    case t.kind == "op" && t.text == "|":
        a := p.next()
        if a.kind != "id" {
            p.fail(a, "|…| takes the name of a slice")
        }
        p.expect("|")
        return &expr{op: opLen, args: []*expr{{op: opVar, name: a.text, sort: sArr, pos: a.pos}}, pos: t.pos}
    case t.kind == "id":
        return p.named(t)
    case t.kind == "eof":
        p.fail(t, "the formula stops short")
    }
    p.fail(t, "unexpected %q", t.text)
    return nil
}
// named parses what starts with an identifier: a variable, true, len(A),
// A[i], a range A[a:b), or P(x, y).
func (p *reader) named(t lexeme) *expr {
    name := strings.TrimRight(t.text, "'")
    primes := len(t.text) - len(name)
    switch name {
    case "true":
        return &expr{op: opTrue, pos: t.pos}
    case "false":
        return &expr{op: opFalse, pos: t.pos}
    case "Forall", "Exists", "Unique", "in":
        p.fail(t, "%s cannot be used as a name", name)
    }
    v := &expr{op: opVar, name: name, primes: primes, pos: t.pos}
    if p.is("[") {
        p.next()
        if r := p.rangeAfter(v, false); r != nil {
            return r
        }
        idx := p.sum()
        if p.is(":") {
            p.next()
            return p.rangeEnd(v, idx, false)
        }
        p.expect("]")
        v.sort = sArr
        return &expr{op: opSel, args: []*expr{v, idx}, pos: t.pos}
    }
    if p.is("(") {
        p.next()
        if r := p.rangeAfter(v, true); r != nil {
            return r
        }
        var args []*expr
        for !p.is(")") {
            a := p.iff()
            if len(args) == 0 && p.is(":") {
                p.next()
                return p.rangeEnd(v, a, true)
            }
            args = append(args, a)
            if !p.is(")") {
                p.expect(",")
            }
        }
        p.expect(")")
        if name == "len" && len(args) == 1 && args[0].op == opVar {
            args[0].sort = sArr
            return &expr{op: opLen, args: args, pos: t.pos}
        }
        return &expr{op: opCall, name: name, args: args, pos: t.pos}
    }
    return v
}

// rangeAfter handles A[:b) and A[:], where the low end is left out.
func (p *reader) rangeAfter(v *expr, open bool) *expr {
    if !p.is(":") {
        return nil
    }
    p.next()
    return p.rangeEnd(v, nil, open)
}

func (p *reader) rangeEnd(v, lo *expr, open bool) *expr {
    r := &expr{op: opRange, name: v.name, args: []*expr{lo, nil}, pos: v.pos}
    r.open[0] = open
    if !p.is("]") && !p.is(")") {
        r.args[1] = p.sum()
    }
    t := p.next()
    switch {
    case t.kind == "op" && t.text == ")":
        r.open[1] = true
    case t.kind == "op" && t.text == "]":
    default:
        p.fail(t, "a range ends in ] or )")
    }
    return r
}

A quantifier’s domain gets folded into its body, as the hypothesis of a Forall and as a conjunct of an Exists. Past this point every quantifier ranges over all the integers, and there’s only one kind of each. Unique x : P(x) says some x makes P true, and anything that makes P true is that x.

// quantifier parses Forall x in D : body. The body reaches as far as it can.
func (p *reader) quantifier() *expr {
    q := p.next()
    x := p.next()
    if x.kind != "id" || strings.HasSuffix(x.text, "'") {
        p.fail(x, "%s needs a variable", q.text)
    }
    if !p.isWord("in") {
        p.fail(p.peek(), "expected \"in\" and a domain, as in %s %s in [0, n) : …", q.text, x.text)
    }
    p.next()
    v := &expr{op: opVar, name: x.text}
    within := p.domain(v)
    p.expect(":")
    body := p.iff()
    switch q.text {
    case "Forall":
        if within != nil {
            body = mk(opImp, within, body)
        }
        return &expr{op: opForall, name: x.text, args: []*expr{body}, pos: q.pos}
    case "Exists":
        if within != nil {
            body = mk(opAnd, within, body)
        }
        return &expr{op: opExists, name: x.text, args: []*expr{body}, pos: q.pos}
    }
    // Unique x: some x makes the body true, and whatever makes it true is x.
    if within != nil {
        body = mk(opAnd, within, body)
    }
    *p.fresh++
    other := fmt.Sprintf("_u%d", *p.fresh)
    same := &expr{op: opForall, name: other, args: []*expr{
        mk(opImp, subst(body, x.text, &expr{op: opVar, name: other}), mk(opEq, &expr{op: opVar, name: other}, v))}}
    return &expr{op: opExists, name: x.text, args: []*expr{mk(opAnd, body, same)}, pos: q.pos}
}

// domain parses Z or an interval, and returns what it says about v.
func (p *reader) domain(v *expr) *expr {
    t := p.next()
    if t.kind == "id" && t.text == "Z" {
        return nil
    }
    if t.kind == "id" && t.text == "B" {
        p.fail(t, "litgo prove quantifies over integers only")
    }
    if t.kind != "op" || t.text != "[" && t.text != "(" {
        p.fail(t, "expected a domain: Z, or an interval like [0, n)")
    }
    var parts []*expr
    if p.is("...") {
        p.next()
    } else {
        parts = append(parts, mk(bound(t.text == "("), p.sum(), v))
    }
    p.expect(",")
    if p.is("...") {
        p.next()
        if c := p.next(); c.kind != "op" || c.text != ")" && c.text != "]" {
            p.fail(c, "an interval ends in ] or )")
        }
    } else {
        hi := p.sum()
        c := p.next()
        if c.kind != "op" || c.text != ")" && c.text != "]" {
            p.fail(c, "an interval ends in ] or )")
        }
        parts = append(parts, mk(bound(c.text == ")"), v, hi))
    }
    return conj(parts)
}
// names lists the free variables of e, sorted.
func names(e *expr) []string {
    seen := map[string]bool{}
    rewrite(e, func(v *expr) *expr {
        seen[v.name] = true
        return nil
    })
    out := make([]string, 0, len(seen))
    for n := range seen {
        out = append(out, n)
    }
    sort.Strings(out)
    return out
}

Walking a function

internal/vego/verify.go

The verifier runs a function symbolically. A parameter starts out as a symbol that stands for any value at all. An assignment doesn’t overwrite a value. It gives the variable a new one and keeps the old, which is what SSA form does with its subscripts. It’s also exactly what the primes of an annotation need: in a formula that mentions x', the x' is the value now and the x is the one before.

A path through the function is a state: what every variable holds, plus the formulas known to be true on the way here. Those formulas are the hypotheses of whatever has to be proved next.

// binding is a program variable on one path: its kind, and every value it
// has had since it was declared or since the loop around it was cut. The
// last one is its value now; the ones before are what primes reach for.
type binding struct {
    kind vkind
    typ  string
    hist []*expr
}

// state is one path through a function: what the variables hold, and what is
// known to be true.
type state struct {
    vars      map[string]*binding
    scopes    []map[string]*binding // what each open block shadowed
    hyps      []*expr
    preserved []kept
    base      []*expr // the BaseCase conditions passed so far
}

type kept struct {
    a     *annot
    depth int
}

func (s *state) clone() *state {
    c := &state{vars: make(map[string]*binding, len(s.vars)), hyps: s.hyps[:len(s.hyps):len(s.hyps)],
        preserved: s.preserved[:len(s.preserved):len(s.preserved)], base: s.base[:len(s.base):len(s.base)]}
    for n, b := range s.vars {
        c.vars[n] = &binding{b.kind, b.typ, append([]*expr{}, b.hist...)}
    }
    for _, sc := range s.scopes {
        m := make(map[string]*binding, len(sc))
        for n, b := range sc {
            m[n] = b
        }
        c.scopes = append(c.scopes, m)
    }
    return c
}

Blocks have scopes, and a variable declared in an inner block can hide one outside. Each open block remembers what it hid and puts it back when it closes.

func (s *state) push() { s.scopes = append(s.scopes, map[string]*binding{}) }

func (s *state) pop() {
    top := s.scopes[len(s.scopes)-1]
    s.scopes = s.scopes[:len(s.scopes)-1]
    for n, old := range top {
        for f := range s.vars {
            if strings.HasPrefix(f, n+".") {
                delete(s.vars, f)
            }
        }
        if old == nil {
            delete(s.vars, n)
        } else {
            s.vars[n] = old
        }
    }
    for len(s.preserved) > 0 && s.preserved[len(s.preserved)-1].depth > len(s.scopes) {
        s.preserved = s.preserved[:len(s.preserved)-1]
    }
}

func (s *state) popTo(depth int) {
    for len(s.scopes) > depth {
        s.pop()
    }
}

func (s *state) declare(name string, b *binding) {
    if name == "_" {
        return
    }
    if top := s.scopes[len(s.scopes)-1]; top != nil {
        if _, ok := top[name]; !ok {
            top[name] = s.vars[name]
        }
    }
    for f := range s.vars {
        if strings.HasPrefix(f, name+".") {
            delete(s.vars, f)
        }
    }
    s.vars[name] = b
}

An if splits a path in two, and both halves carry on. So a statement doesn’t leave one state behind but several, sorted by how they left: out the bottom, by break, or by continue. A return leaves none, because everything that has to hold at a return is proved right there on the spot.

// outcomes are the states a statement leaves behind, by how it was left.
// A return leaves none: its obligations are settled on the spot.
type outcomes struct {
    normal, brk, cont []*state
}

func (o *outcomes) add(p outcomes) {
    o.normal = append(o.normal, p.normal...)
    o.brk = append(o.brk, p.brk...)
    o.cont = append(o.cont, p.cont...)
}
// run verifies one function.
type run struct {
    v        *verifier
    f        *fn
    entry    map[string]*expr // the parameters' values on entry
    blocks   map[*ast.BlockStmt][]*annot
    measure0 []*expr
    paths    int
    depth    int // of one-line functions being unfolded
}

// unsupported is how the verifier backs out of a function it cannot follow.
type unsupported struct {
    pos token.Pos
    msg string
}

func (r *run) no(n ast.Node, format string, args ...any) {
    panic(unsupported{n.Pos(), fmt.Sprintf(format, args...)})
}

const maxPaths = 400

Verifying a function starts with its parameters as unknowns and its precondition as the first hypothesis. Go gives named results their zero values, so the proof knows those too.

func (v *verifier) verify(f *fn) {
    r := &run{v: v, f: f, entry: map[string]*expr{}, blocks: map[*ast.BlockStmt][]*annot{}}
    defer func() {
        if p := recover(); p != nil {
            u, ok := p.(unsupported)
            if !ok {
                // A bug in the verifier. The editor is on the other end of this,
                // so say so at the function instead of taking the server down.
                u = unsupported{f.decl.Name.Pos(), fmt.Sprintf("the verifier failed (%v); please report this", p)}
            }
            at := v.position(u.pos)
            v.report(at, 0, false, fmt.Sprintf("%s is not verified: %s", f.key, u.msg))
        }
    }()
    r.place()
    st := &state{vars: map[string]*binding{}}
    st.push()
    all := append([]param{}, f.params...)
    if f.recv != nil {
        all = append(all, *f.recv)
    }
    for _, p := range all {
        if p.kind == kNone && p.name != "_" {
            // Whatever it is, the proof cannot mention it.
            continue
        }
        b := &binding{kind: p.kind, typ: p.typ}
        if p.kind != kStruct {
            b.hist = []*expr{r.fresh(p.name, p.kind)}
            r.entry[p.name] = b.hist[0]
        }
        st.declare(p.name, b)
    }
    for _, p := range f.results {
        if p.kind != kInt && p.kind != kBool {
            r.no(f.decl, "its result %s is neither an integer nor a boolean", p.name)
        }
        zero := num(0)
        if p.kind == kBool {
            zero = eFalse
        }
        st.declare(p.name, &binding{kind: p.kind, hist: []*expr{zero}})
    }
    for _, a := range f.requires {
        if a.e != nil {
            st.hyps = append(st.hyps, r.formula(st, a, false))
        }
    }
    for _, a := range f.measure {
        if a.e != nil {
            r.measure0 = append(r.measure0, r.formula(st, a, false))
        }
    }
    r.lint()
    if r.recursive() && len(f.measure) == 0 {
        v.report(v.position(f.decl.Name.Pos()), 0, true, f.key+" calls itself and has no Measure, so nothing shows that it terminates")
    }
    out := r.block(st, f.decl.Body)
    for _, s := range out.normal {
        r.leave(s, f.decl.Body.Rbrace)
    }
}
// place sorts the annotations of the body into the blocks they are written in.
func (r *run) place() {
    for _, a := range r.f.inside {
        var inner *ast.BlockStmt
        ast.Inspect(r.f.decl.Body, func(n ast.Node) bool {
            if b, ok := n.(*ast.BlockStmt); ok && a.pos > b.Lbrace && a.pos < b.Rbrace {
                inner = b
            }
            return true
        })
        if inner != nil {
            r.blocks[inner] = append(r.blocks[inner], a)
        }
    }
}

// lint points at contracts that say nothing.
func (r *run) lint() {
    for _, list := range [][]*annot{r.f.requires, r.f.ensures} {
        for _, a := range list {
            if a.e != nil && a.e.op == opTrue {
                r.v.report(a.where, a.endCol, true, a.kind+" true says nothing")
            }
        }
    }
}

func (r *run) recursive() bool {
    found := false
    ast.Inspect(r.f.decl.Body, func(n ast.Node) bool {
        if c, ok := n.(*ast.CallExpr); ok {
            if callee, _ := r.callee(nil, c); callee == r.f {
                found = true
            }
        }
        return true
    })
    return found
}

func (r *run) fresh(name string, k vkind) *expr {
    r.v.fresh++
    s := sInt
    if k == kBool {
        s = sBool
    } else if k == kSlice {
        s = sArr
    }
    return sym(fmt.Sprintf("%s#%d", name, r.v.fresh), s)
}

Obligations

Everything the verifier proves goes through check: the path’s hypotheses on one side, a goal on the other. A failure is reported at the annotation that wasn’t proved, and names the place in the code it failed for. An obligation that comes from the code itself, like an index, is reported at the code.

// check is one proof obligation. It reports at the annotation, or at a
// place in the code, and mentions a second place if there is one.
func (r *run) check(st *state, goal *expr, a *annot, at ast.Node, ref ast.Node, msg string) bool {
    r.f.obligations++
    if proves(st.hyps, goal) {
        return true
    }
    where, end := Pos{}, 0
    if a != nil {
        where, end = a.where, a.endCol
    } else {
        where = r.v.position(at.Pos())
        end = where.Col + int(at.End()-at.Pos())
    }
    var rp Pos
    if ref != nil {
        rp = r.v.position(ref.Pos())
    }
    r.v.reportRef(where, end, false, msg, rp)
    return false
}

func (r *run) assume(st *state, f *expr) { st.hyps = append(st.hyps, f) }

// feasible reports whether a path can be taken at all. Pruning the ones that
// cannot keeps their obligations from being reported as failures of logic.
func (r *run) feasible(st *state) bool { return !proves(st.hyps, eFalse) }

What a formula means at a point in the program

A formula is written in the program’s names, and what it says depends on where it stands. This is where the names get their values. With primes, the most-primed mention of a variable is its value now, and each prime fewer is one assignment earlier. So after x = x + 1, the formula x' = x + 1 says exactly what the assignment did. A postcondition is different. There, a parameter without a prime is the value the caller passed in, whatever the body did to its copy.

// formula gives an annotation's formula its meaning in a state: predicates
// and one-line functions unfolded, names replaced by values. In a
// postcondition a parameter means its value on entry, and primed, its value
// at the end.
func (r *run) formula(st *state, a *annot, post bool) *expr {
    e := r.unfold(st, a, a.e, 0)
    most := map[string]int{}
    var count func(e *expr)
    count = func(e *expr) {
        if e == nil {
            return
        }
        if e.op == opVar {
            most[e.name] = max(most[e.name], e.primes)
        }
        for _, x := range e.args {
            count(x)
        }
    }
    count(e)
    return rewrite(e, func(v *expr) *expr {
        if strings.ContainsAny(v.name, "#?") {
            return nil
        }
        if old, ok := r.entry[v.name]; ok && post {
            if v.primes == 0 {
                return old
            }
            return r.lookup(st, a, v.name, 0)
        }
        return r.lookup(st, a, v.name, most[v.name]-v.primes)
    })
}

// lookup finds the value a name had `back` assignments ago.
func (r *run) lookup(st *state, a *annot, name string, back int) *expr {
    b := r.binding(st, name)
    if b == nil {
        if c, ok := r.v.consts[name]; ok {
            return num(c)
        }
        r.bad(a, "%s is not a variable here", name)
    }
    if b.kind == kStruct || len(b.hist) == 0 {
        r.bad(a, "%s is not an integer, a boolean or a slice of integers", name)
    }
    if back >= len(b.hist) {
        r.bad(a, "%s has not been assigned often enough on this path for that many primes", name)
    }
    return b.hist[len(b.hist)-1-back]
}
// binding finds a variable, or makes one for the field of a struct the
// first time it is mentioned. Nothing in the fragment assigns to a field, so
// a field holds one value throughout.
func (r *run) binding(st *state, name string) *binding {
    if b := st.vars[name]; b != nil {
        return b
    }
    base, field, ok := strings.Cut(name, ".")
    if !ok {
        return nil
    }
    s := st.vars[base]
    if s == nil || s.kind != kStruct {
        return nil
    }
    var k vkind
    switch r.v.structs[s.typ][field] {
    case "int", "int8", "int16", "int32", "int64":
        k = kInt
    case "bool":
        k = kBool
    case "[]int":
        k = kSlice
    default:
        return nil
    }
    b := &binding{kind: k, hist: []*expr{r.fresh(name, k)}}
    st.vars[name] = b
    return b
}

// bad gives up on an annotation that does not make sense.
func (r *run) bad(a *annot, format string, args ...any) {
    msg := fmt.Sprintf(format, args...)
    if a == nil {
        panic(unsupported{r.f.decl.Pos(), msg})
    }
    r.v.report(a.where, a.endCol, false, "in this "+a.kind+": "+msg)
    panic(unsupported{r.f.decl.Name.Pos(), "one of its annotations does not make sense"})
}

A predicate gets unfolded into its body. So does a call of a Go function whose whole body is return plus one expression, which turns small Go functions into vocabulary: a contract can say s.src(c) = col and mean it about the real src. A function with more going on can’t be used this way, because a formula needs a value and such a function might never even return.

// unfold replaces P(x, y) by what it stands for: the body of a predicate, or
// the expression a one-line Go function returns.
func (r *run) unfold(st *state, a *annot, e *expr, depth int) *expr {
    if e == nil {
        return nil
    }
    if depth > 12 {
        r.bad(a, "%s is defined in terms of itself", e.name)
    }
    c := *e
    c.args = make([]*expr, len(e.args))
    for i, x := range e.args {
        c.args[i] = r.unfold(st, a, x, depth)
    }
    if e.op != opCall {
        return &c
    }
    if p := r.v.preds[e.name]; p != nil {
        if len(p.params) != len(c.args) {
            r.bad(a, "%s takes %d arguments", p.name, len(p.params))
        }
        return r.unfold(st, a, replace(p.body, p.params, c.args, "", ""), depth+1)
    }
    callee, recv := r.named(st, e.name)
    if callee == nil {
        r.bad(a, "%s is neither a predicate nor a function of this package", e.name)
    }
    body := r.oneLine(callee)
    if body == nil {
        r.bad(a, "%s cannot be used in a formula: only a function whose body is one return of one expression can", e.name)
    }
    if len(callee.params) != len(c.args) {
        r.bad(a, "%s takes %d arguments", e.name, len(callee.params))
    }
    var formals []string
    for _, p := range callee.params {
        formals = append(formals, p.name)
    }
    from := ""
    if callee.recv != nil {
        from = callee.recv.name
    }
    return r.unfold(st, a, replace(body, formals, c.args, from, recv), depth+1)
}

// replace substitutes all the parameters at once, and renames the fields of
// the callee's receiver to fields of the caller's.
func replace(body *expr, formals []string, actuals []*expr, from, to string) *expr {
    return rewrite(body, func(v *expr) *expr {
        for i, f := range formals {
            if v.name == f && v.primes == 0 {
                return actuals[i]
            }
        }
        if from != "" && strings.HasPrefix(v.name, from+".") {
            c := *v
            c.name = to + strings.TrimPrefix(v.name, from)
            return &c
        }
        return nil
    })
}

// named finds the function a formula or a call mentions: f, or x.m where x
// is a struct.
func (r *run) named(st *state, name string) (*fn, string) {
    if f := r.v.funcs[name]; f != nil && f.recv == nil {
        return f, ""
    }
    if i := strings.LastIndex(name, "."); i > 0 && st != nil {
        if b := st.vars[name[:i]]; b != nil && b.kind == kStruct {
            return r.v.funcs[b.typ+name[i:]], name[:i]
        }
    }
    return nil, ""
}

// oneLine is the expression a function returns, if that is all it does.
func (r *run) oneLine(f *fn) *expr {
    if len(f.decl.Body.List) != 1 || len(f.results) != 1 || len(f.inside) > 0 {
        return nil
    }
    ret, ok := f.decl.Body.List[0].(*ast.ReturnStmt)
    if !ok || len(ret.Results) != 1 {
        return nil
    }
    kinds := map[string]vkind{}
    for _, p := range f.params {
        kinds[p.name] = p.kind
    }
    var e *expr
    func() {
        defer func() {
            if p := recover(); p != nil {
                if _, ok := p.(unsupported); !ok {
                    panic(p)
                }
                e = nil
            }
        }()
        e = r.pure(ret.Results[0], func(name string) vkind {
            if k, ok := kinds[name]; ok {
                return k
            }
            if f.recv != nil && strings.HasPrefix(name, f.recv.name+".") {
                if r.v.structs[f.recv.typ][strings.TrimPrefix(name, f.recv.name+".")] == "bool" {
                    return kBool
                }
            }
            return kInt
        })
    }()
    return e
}

Go expressions

A Go expression is read in two steps. pure translates the syntax into a term over the program’s names, remembering where every part came from. eval then gives the names their values and, along the way, states what has to be true for the expression to be evaluated at all.

// pure reads a Go expression as a term over the names it mentions. It
// checks nothing yet; pos remembers where each part came from.
func (r *run) pure(e ast.Expr, kind func(string) vkind) *expr {
    at := func(x *expr) *expr { x.pos = int(e.Pos()); return x }
    switch e := e.(type) {
    case *ast.ParenExpr:
        return r.pure(e.X, kind)
    case *ast.BasicLit:
        if c, ok := constInt(e); ok {
            return at(num(c))
        }
    case *ast.Ident:
        switch e.Name {
        case "true":
            return eTrue
        case "false":
            return eFalse
        }
        s := sInt
        if kind(e.Name) == kBool {
            s = sBool
        } else if kind(e.Name) == kSlice {
            s = sArr
        }
        return at(sym(e.Name, s))
    case *ast.SelectorExpr:
        if x, ok := e.X.(*ast.Ident); ok {
            name := x.Name + "." + e.Sel.Name
            s := sInt
            if kind(name) == kBool {
                s = sBool
            } else if kind(name) == kSlice {
                s = sArr
            }
            return at(sym(name, s))
        }
    case *ast.UnaryExpr:
        x := r.pure(e.X, kind)
        switch e.Op {
        case token.SUB:
            return at(mk(opNeg, x))
        case token.ADD:
            return x
        case token.NOT:
            return at(mk(opNot, x))
        }
    case *ast.BinaryExpr:
        l, rt := r.pure(e.X, kind), r.pure(e.Y, kind)
        switch e.Op {
        case token.ADD:
            return at(mk(opAdd, l, rt))
        case token.SUB:
            return at(mk(opSub, l, rt))
        case token.MUL:
            return at(mk(opMul, l, rt))
        case token.QUO:
            return at(mk(opDiv, l, rt))
        case token.REM:
            return at(mk(opMod, l, rt))
        case token.LAND:
            return at(mk(opAnd, l, rt))
        case token.LOR:
            return at(mk(opOr, l, rt))
        case token.LSS:
            return at(mk(opLt, l, rt))
        case token.LEQ:
            return at(mk(opLe, l, rt))
        case token.GTR:
            return at(mk(opLt, rt, l))
        case token.GEQ:
            return at(mk(opLe, rt, l))
        case token.EQL, token.NEQ:
            var c *expr
            if isBool(l) || isBool(rt) {
                c = mk(opIff, l, rt)
            } else {
                c = mk(opEq, l, rt)
            }
            if e.Op == token.NEQ {
                c = mk(opNot, c)
            }
            return at(c)
        }
    case *ast.IndexExpr:
        a := r.pure(e.X, kind)
        if a.op == opVar {
            a.sort = sArr
            return at(mk(opSel, a, r.pure(e.Index, kind)))
        }
    case *ast.CallExpr:
        name := ""
        switch f := e.Fun.(type) {
        case *ast.Ident:
            name = f.Name
        case *ast.SelectorExpr:
            if x, ok := f.X.(*ast.Ident); ok {
                name = x.Name + "." + f.Sel.Name
            }
        }
        if name == "" {
            break
        }
        var args []*expr
        for _, x := range e.Args {
            args = append(args, r.pure(x, kind))
        }
        if name == "len" && len(args) == 1 && args[0].op == opVar {
            args[0].sort = sArr
            return at(mk(opLen, args[0]))
        }
        if name == "int" && len(args) == 1 {
            return args[0]
        }
        c := at(&expr{op: opCall, name: name, args: args})
        return c
    }
    r.no(e, "this expression is outside what litgo prove understands (integers, booleans, slices of integers, fields of structs, calls of verified functions)")
    return nil
}

These are the safety obligations: an index stays inside its slice, a divisor isn’t zero, a callee’s precondition holds. Go evaluates && and || left to right and stops early, so the right side of i < len(A) && A[i] > 0 only has to be safe when the left side is true. The guards carry that.

// value evaluates a Go expression in a state, and states what has to be
// true for the evaluation to be safe: indices in range, divisors not zero,
// the preconditions of what is called.
func (r *run) value(st *state, e ast.Expr) *expr {
    p := r.pure(e, func(name string) vkind {
        if b := r.binding(st, name); b != nil {
            return b.kind
        }
        return kNone
    })
    vals := r.eval(st, p, nil, e)
    if len(vals) != 1 {
        r.no(e, "a call with %d results cannot be used as a value", len(vals))
    }
    return vals[0]
}

// eval does the work of value. The guards are what is known at this point
// of the expression and not before: the left side of an && while its right
// side is being evaluated.
func (r *run) eval(st *state, e *expr, guards []*expr, at ast.Node) []*expr {
    one := func(x *expr) *expr {
        vals := r.eval(st, x, guards, at)
        if len(vals) != 1 {
            r.no(at, "a call with %d results cannot be used as a value", len(vals))
        }
        return vals[0]
    }
    guarded := func(goal *expr) *expr {
        if len(guards) == 0 {
            return goal
        }
        return mk(opImp, conj(guards), goal)
    }
    here := ast.Node(at)
    if e.pos != 0 {
        here = point(e.pos)
    }
    switch e.op {
    case opNum, opTrue, opFalse:
        return []*expr{e}
    case opVar:
        if strings.ContainsAny(e.name, "#?") {
            return []*expr{e}
        }
        b := r.binding(st, e.name)
        if b == nil {
            if c, ok := r.v.consts[e.name]; ok {
                return []*expr{num(c)}
            }
            r.no(here, "%s is not a variable the proof can talk about", e.name)
        }
        if len(b.hist) == 0 {
            r.no(here, "%s is a struct, and only its fields have values", e.name)
        }
        return []*expr{b.hist[len(b.hist)-1]}
    case opAnd, opOr:
        l := one(e.args[0])
        g := l
        if e.op == opOr {
            g = not(l)
        }
        rt := r.eval(st, e.args[1], append(guards[:len(guards):len(guards)], g), at)
        return []*expr{mk(e.op, l, rt[0])}
    case opSel:
        arr, idx := one(e.args[0]), one(e.args[1])
        in := mk(opAnd, mk(opLe, num(0), idx), mk(opLt, idx, mk(opLen, arr)))
        r.check(st, guarded(in), nil, here, nil, "the index may be out of range: 0 <= "+e.args[1].String()+" < len("+e.args[0].String()+") is not proved")
        r.assume(st, guarded(in))
        return []*expr{mk(opSel, arr, idx)}
    case opDiv, opMod:
        l, d := one(e.args[0]), one(e.args[1])
        r.check(st, guarded(mk(opNe, d, num(0))), nil, here, nil, "the divisor may be zero: "+e.args[1].String()+" <> 0 is not proved")
        r.assume(st, guarded(mk(opNe, d, num(0))))
        return []*expr{mk(e.op, l, d)}
    case opCall:
        var args []*expr
        for _, x := range e.args {
            args = append(args, one(x))
        }
        return r.call(st, e, args, guards, here)
    }
    c := *e
    c.args = make([]*expr, len(e.args))
    for i, x := range e.args {
        c.args[i] = one(x)
    }
    return []*expr{&c}
}
// point is a position that stands in for a node.
type point token.Pos

func (p point) Pos() token.Pos { return token.Pos(p) }
func (p point) End() token.Pos { return token.Pos(p) + 1 }

// callee finds the declared function a call expression calls.
func (r *run) callee(st *state, c *ast.CallExpr) (*fn, string) {
    switch f := c.Fun.(type) {
    case *ast.Ident:
        if fn := r.v.funcs[f.Name]; fn != nil && fn.recv == nil {
            return fn, ""
        }
    case *ast.SelectorExpr:
        x, ok := f.X.(*ast.Ident)
        if !ok {
            return nil, ""
        }
        if st == nil { // asked only whether the function calls itself
            if r.f.recv != nil && x.Name == r.f.recv.name && r.f.key == r.f.recv.typ+"."+f.Sel.Name {
                return r.f, x.Name
            }
            return nil, ""
        }
        return r.named(st, x.Name+"."+f.Sel.Name)
    }
    return nil, ""
}

A call is where verification becomes modular. The caller never looks inside the callee. It proves the callee’s Requires, gets fresh unknowns for the results, and learns the callee’s Ensures about them, and nothing else. That’s also what makes recursion work. Inside Triangle, the call Triangle(n-1) is known by the very contract being proved. That’s the induction hypothesis, and the Measure is what keeps the induction well-founded (it can’t go down forever).

// call uses a function by its contract: the precondition is an obligation,
// the postcondition is what is known about the results. A function that is
// one expression is used as that expression, which says more.
func (r *run) call(st *state, e *expr, args []*expr, guards []*expr, at ast.Node) []*expr {
    switch e.name {
    case "min", "max":
        if len(args) != 2 || r.v.funcs[e.name] != nil {
            break
        }
        m := r.fresh(e.name, kInt)
        lo, hi := m, args[0]
        lo2, hi2 := m, args[1]
        if e.name == "max" {
            lo, hi, lo2, hi2 = args[0], m, args[1], m
        }
        r.assume(st, conj([]*expr{mk(opLe, lo, hi), mk(opLe, lo2, hi2), mk(opOr, mk(opEq, m, args[0]), mk(opEq, m, args[1]))}))
        return []*expr{m}
    }
    callee, recv := r.named(st, e.name)
    if callee == nil {
        r.no(at, "%s has no contract the proof could use; only functions of this package can be called", e.name)
    }
    if len(args) != len(callee.params) {
        r.no(at, "%s takes %d arguments", e.name, len(callee.params))
    }
    if !callee.annotated && r.oneLine(callee) == nil {
        // Nobody has checked what it does to the slices and structs it can reach.
        r.no(at, "%s has no annotations, so nothing is known about what it does; give it a contract, even an empty one like Requires true", e.name)
    }
    guarded := func(goal *expr) *expr {
        if len(guards) == 0 {
            return goal
        }
        return mk(opImp, conj(guards), goal)
    }
    // The callee's formulas, in the caller's terms.
    inst := func(a *annot, results []*expr) *expr {
        if a.e == nil {
            return eTrue
        }
        sub := &state{vars: map[string]*binding{}}
        sub.push()
        for i, p := range callee.params {
            if p.kind == kNone {
                continue
            }
            sub.vars[p.name] = &binding{kind: p.kind, typ: p.typ, hist: []*expr{args[i]}}
            if results != nil && p.kind != kSlice {
                // Primed, a parameter is the callee's own copy when it returned,
                // and the caller knows nothing about that.
                sub.vars[p.name].hist = []*expr{r.fresh(p.name, p.kind)}
            }
        }
        if callee.recv != nil && callee.recv.kind == kStruct {
            sub.vars[callee.recv.name] = &binding{kind: kStruct, typ: callee.recv.typ}
            for field := range r.v.structs[callee.recv.typ] {
                if b := r.binding(st, recv+"."+field); b != nil {
                    sub.vars[callee.recv.name+"."+field] = b
                }
            }
        }
        for i, p := range callee.results {
            if results != nil {
                sub.vars[p.name] = &binding{kind: p.kind, hist: []*expr{results[i]}}
            }
        }
        inner := &run{v: r.v, f: callee, entry: map[string]*expr{}}
        for i, p := range callee.params {
            if p.kind != kNone && p.kind != kStruct {
                inner.entry[p.name] = args[i]
            }
        }
        defer func() {
            // What goes wrong in the callee's annotation is the callee's problem.
            if p := recover(); p != nil {
                if _, ok := p.(unsupported); !ok {
                    panic(p)
                }
                r.no(at, "the contract of %s cannot be used here", e.name)
            }
        }()
        return inner.formula(sub, a, results != nil)
    }
    for _, a := range callee.requires {
        r.check(st, guarded(inst(a, nil)), nil, at, nil, fmt.Sprintf("this call does not establish what %s requires: %s", callee.key, a.text))
    }
    if callee == r.f {
        r.descends(st, e, args, guards, at)
    }
    if body := r.oneLine(callee); body != nil {
        if r.depth++; r.depth > 40 {
            r.no(at, "%s never stops calling itself", callee.key)
        }
        defer func() { r.depth-- }()
        var formals []string
        for _, p := range callee.params {
            formals = append(formals, p.name)
        }
        from := ""
        if callee.recv != nil {
            from = callee.recv.name
        }
        return r.eval(st, replace(body, formals, args, from, recv), guards, at)
    }
    var results []*expr
    for _, p := range callee.results {
        results = append(results, r.fresh(callee.decl.Name.Name+"."+p.name, p.kind))
    }
    for _, a := range callee.ensures {
        r.assume(st, guarded(inst(a, results)))
    }
    for _, a := range callee.exsures {
        r.assume(st, guarded(not(inst(a, results))))
    }
    return results
}
// descends checks a recursive call: the measure is smaller than it was on
// entry and was not negative, and no base case has been passed.
func (r *run) descends(st *state, e *expr, args []*expr, guards []*expr, at ast.Node) {
    guarded := func(goal *expr) *expr {
        if len(guards) == 0 {
            return goal
        }
        return mk(opImp, conj(guards), goal)
    }
    for i, a := range r.f.measure {
        if a.e == nil {
            continue
        }
        sub := &state{vars: map[string]*binding{}}
        for j, p := range r.f.params {
            if p.kind != kNone && p.kind != kStruct {
                sub.vars[p.name] = &binding{kind: p.kind, hist: []*expr{args[j]}}
            }
        }
        m := (&run{v: r.v, f: r.f, entry: map[string]*expr{}}).formula(sub, a, false)
        r.check(st, guarded(mk(opLt, m, r.measure0[i])), a, nil, at, "Measure is not shown to get smaller in the recursive call at {ref}")
        r.check(st, guarded(mk(opLe, num(0), r.measure0[i])), a, nil, at, "Measure can be negative when the recursive call at {ref} is made")
    }
    for _, b := range st.base {
        r.check(st, guarded(not(b)), nil, at, nil, "this recursive call can be reached in the base case")
    }
}

Statements

A block is its statements, with the annotations between them taken in order. Invariant and Variant are the exception: the loop they belong to collects them.

// block runs the statements of a block, with the annotations between them.
func (r *run) block(st *state, b *ast.BlockStmt) outcomes {
    st.push()
    depth := len(st.scopes)
    var out outcomes
    live := []*state{st}
    notes := r.blocks[b]
    n := 0
    for i := 0; i <= len(b.List); i++ {
        var next ast.Stmt
        limit := b.Rbrace
        if i < len(b.List) {
            next = b.List[i]
            limit = next.Pos()
        }
        var pending []*annot
        for n < len(notes) && notes[n].pos < limit {
            a := notes[n]
            n++
            if a.kind == "Invariant" || a.kind == "Variant" {
                if isLoop(next) {
                    pending = append(pending, a)
                }
                continue // at the end of a loop's body it belongs to that loop
            }
            for _, s := range live {
                r.note(s, a)
            }
        }
        if next == nil {
            break
        }
        var after []*state
        for _, s := range live {
            o := r.stmt(s, next, pending)
            after = append(after, o.normal...)
            out.brk = append(out.brk, o.brk...)
            out.cont = append(out.cont, o.cont...)
        }
        live = after
        if r.paths += len(live); r.paths > maxPaths*50 || len(live) > maxPaths {
            r.no(next, "there are too many paths through it")
        }
    }
    for _, s := range live {
        s.popTo(depth - 1)
    }
    out.normal = live
    return out
}

func isLoop(s ast.Stmt) bool {
    switch s.(type) {
    case *ast.ForStmt, *ast.RangeStmt:
        return true
    }
    return false
}
// note handles an annotation that stands where a statement could.
func (r *run) note(st *state, a *annot) {
    a.used = true
    if a.e == nil {
        return
    }
    switch a.kind {
    case "Assert", "InductionHypothesis":
        f := r.formula(st, a, false)
        r.check(st, f, a, nil, nil, a.kind+" is not proved: "+a.text)
        r.assume(st, f)
    case "Axiom":
        r.v.report(a.where, a.endCol, true, "Axiom is taken on trust, not proved: "+a.text)
        r.assume(st, r.formula(st, a, false))
    case "Preserves":
        f := r.formula(st, a, false)
        r.check(st, f, a, nil, nil, "Preserves does not hold to begin with: "+a.text)
        r.assume(st, f)
        st.preserved = append(st.preserved, kept{a, len(st.scopes)})
    case "BaseCase":
        st.base = append(st.base, r.formula(st, a, false))
    case "Immutable":
    default:
        r.v.report(a.where, a.endCol, false, a.kind+" does not belong inside a function body")
    }
}
func (r *run) stmt(st *state, s ast.Stmt, pending []*annot) outcomes {
    switch s := s.(type) {
    case *ast.EmptyStmt:
        return outcomes{normal: []*state{st}}
    case *ast.BlockStmt:
        return r.block(st, s)
    case *ast.DeclStmt:
        g, ok := s.Decl.(*ast.GenDecl)
        if !ok || g.Tok != token.VAR {
            r.no(s, "only variables can be declared inside a verified function")
        }
        for _, spec := range g.Specs {
            vs := spec.(*ast.ValueSpec)
            for i, n := range vs.Names {
                var val *expr
                k := kNone
                if vs.Type != nil {
                    k, _ = r.v.kindOf(vs.Type)
                }
                if i < len(vs.Values) {
                    val = r.value(st, vs.Values[i])
                    if k == kNone {
                        k = kInt
                        if isBool(val) {
                            k = kBool
                        }
                    }
                } else if k == kInt {
                    val = num(0)
                } else if k == kBool {
                    val = eFalse
                }
                if val == nil || k != kInt && k != kBool {
                    r.no(s, "%s is neither an integer nor a boolean", n.Name)
                }
                st.declare(n.Name, &binding{kind: k, hist: []*expr{val}})
            }
        }
        return outcomes{normal: []*state{st}}
    case *ast.AssignStmt:
        r.assign(st, s)
        return outcomes{normal: []*state{st}}
    case *ast.IncDecStmt:
        one := mk(opAdd, r.value(st, s.X), num(1))
        if s.Tok == token.DEC {
            one.op = opSub
        }
        r.set(st, s.X, one, s)
        return outcomes{normal: []*state{st}}
    case *ast.ExprStmt:
        r.effect(st, s)
        return outcomes{normal: []*state{st}}
    case *ast.ReturnStmt:
        r.ret(st, s)
        return outcomes{}
    case *ast.BranchStmt:
        if s.Label != nil {
            r.no(s, "labels are outside what litgo prove understands")
        }
        switch s.Tok {
        case token.BREAK:
            return outcomes{brk: []*state{st}}
        case token.CONTINUE:
            return outcomes{cont: []*state{st}}
        }
    case *ast.IfStmt:
        return r.branch(st, s)
    case *ast.ForStmt:
        return r.loop(st, &loop{stmt: s, init: s.Init, cond: s.Cond, post: s.Post, body: s.Body}, pending)
    case *ast.RangeStmt:
        return r.loop(st, &loop{stmt: s, over: s, body: s.Body}, pending)
    }
    r.no(s, "this statement is outside what litgo prove understands")
    return outcomes{}
}
// effect is a call made for what it does. A call of a verified function is
// checked like any other. A call of anything else is let through if all it
// is handed are numbers, booleans and literals, because then it cannot
// change anything the proof is about. panic ends the path.
func (r *run) effect(st *state, s *ast.ExprStmt) {
    c, ok := s.X.(*ast.CallExpr)
    if !ok {
        r.no(s, "this statement is outside what litgo prove understands")
    }
    if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "panic" {
        r.assume(st, eFalse)
        return
    }
    if callee, _ := r.callee(st, c); callee != nil {
        r.eval(st, r.pure(c, func(name string) vkind {
            if b := r.binding(st, name); b != nil {
                return b.kind
            }
            return kNone
        }), nil, c)
        return
    }
    if sel, ok := c.Fun.(*ast.SelectorExpr); ok {
        if x, ok := sel.X.(*ast.Ident); ok && st.vars[x.Name] != nil {
            r.no(c, "%s.%s has no contract, and a method may change what it is called on", x.Name, sel.Sel.Name)
        }
    }
    for _, a := range c.Args {
        if lit, ok := a.(*ast.BasicLit); ok && lit.Kind != token.INT {
            continue
        }
        if v := r.value(st, a); v.sort == sArr {
            r.no(a, "a slice handed to a function without a contract may come back changed")
        }
    }
}

Assignments go to variables only. A[i] = x and p.f = x are refused, and the reason is aliasing. Another slice may share the array, another pointer may reach the struct, and a proof that didn’t see them change would be a proof of something false.

func (r *run) assign(st *state, s *ast.AssignStmt) {
    ops := map[token.Token]op{token.ADD_ASSIGN: opAdd, token.SUB_ASSIGN: opSub, token.MUL_ASSIGN: opMul,
        token.QUO_ASSIGN: opDiv, token.REM_ASSIGN: opMod}
    if o, ok := ops[s.Tok]; ok {
        bin := &ast.BinaryExpr{X: s.Lhs[0], OpPos: s.TokPos, Y: s.Rhs[0],
            Op: map[op]token.Token{opAdd: token.ADD, opSub: token.SUB, opMul: token.MUL, opDiv: token.QUO, opMod: token.REM}[o]}
        r.set(st, s.Lhs[0], r.value(st, bin), s)
        return
    }
    if s.Tok != token.ASSIGN && s.Tok != token.DEFINE {
        r.no(s, "this assignment is outside what litgo prove understands")
    }
    var vals []*expr
    if len(s.Rhs) == 1 && len(s.Lhs) > 1 {
        c, ok := s.Rhs[0].(*ast.CallExpr)
        if !ok {
            r.no(s, "only a call can give several values at once")
        }
        vals = r.eval(st, r.pure(c, func(name string) vkind {
            if b := r.binding(st, name); b != nil {
                return b.kind
            }
            return kNone
        }), nil, c)
    } else {
        for _, x := range s.Rhs {
            vals = append(vals, r.value(st, x))
        }
    }
    if len(vals) != len(s.Lhs) {
        r.no(s, "the two sides of this assignment do not match")
    }
    for i, l := range s.Lhs {
        id, ok := l.(*ast.Ident)
        if s.Tok == token.DEFINE && ok && (st.vars[id.Name] == nil || !r.inScope(st, id.Name)) {
            k := kInt
            if isBool(vals[i]) {
                k = kBool
            } else if vals[i].sort == sArr {
                r.no(s, "slices cannot be copied inside a verified function")
            }
            st.declare(id.Name, &binding{kind: k, hist: []*expr{vals[i]}})
            continue
        }
        r.store(st, l, vals[i])
    }
    // A parallel assignment is one step: what is preserved is looked at
    // once all of it has happened.
    r.keep(st, s)
}

// inScope reports whether a name was declared in the innermost block, which
// is what decides whether := declares it again.
func (r *run) inScope(st *state, name string) bool {
    _, ok := st.scopes[len(st.scopes)-1][name]
    return ok
}

// set assigns to a variable, and then looks at what was promised to be
// preserved.
func (r *run) set(st *state, target ast.Expr, val *expr, at ast.Node) {
    r.store(st, target, val)
    r.keep(st, at)
}

func (r *run) store(st *state, target ast.Expr, val *expr) {
    id, ok := target.(*ast.Ident)
    if !ok {
        r.no(target, "only variables can be assigned to: an element of a slice or a field may have other names, and the proof would not see them change")
    }
    if id.Name == "_" {
        return
    }
    b := st.vars[id.Name]
    if b == nil || b.kind != kInt && b.kind != kBool {
        r.no(target, "%s is not an integer or boolean variable", id.Name)
    }
    b.hist = append(b.hist, val)
}

func (r *run) keep(st *state, at ast.Node) {
    for _, k := range st.preserved {
        f := r.formula(st, k.a, false)
        r.check(st, f, k.a, nil, at, "Preserves is broken by the assignment at {ref}: "+k.a.text)
        r.assume(st, f)
    }
}

A branch that can’t be taken gets dropped, by asking the prover whether the hypotheses have become contradictory. Without that, the obligations of unreachable code would be “proved” from the contradiction. Harmless, but paths multiply, and these are the cheap ones to lose.

func (r *run) branch(st *state, s *ast.IfStmt) outcomes {
    st.push()
    depth := len(st.scopes)
    var out outcomes
    starts := []*state{st}
    if s.Init != nil {
        starts = r.stmt(st, s.Init, nil).normal
    }
    for _, st := range starts {
        cond := r.value(st, s.Cond)
        yes, no := st, st.clone()
        r.assume(yes, cond)
        r.assume(no, not(cond))
        if r.feasible(yes) {
            out.add(r.block(yes, s.Body))
        }
        if r.feasible(no) {
            switch e := s.Else.(type) {
            case nil:
                out.normal = append(out.normal, no)
            case *ast.BlockStmt:
                out.add(r.block(no, e))
            case *ast.IfStmt:
                out.add(r.branch(no, e))
            }
        }
    }
    for _, s := range out.normal {
        s.popTo(depth - 1)
    }
    return out
}
// ret settles a return: the results get their values, and every
// postcondition is an obligation.
func (r *run) ret(st *state, s *ast.ReturnStmt) {
    if len(s.Results) > 0 {
        var vals []*expr
        if len(s.Results) == 1 && len(r.f.results) > 1 {
            c, ok := s.Results[0].(*ast.CallExpr)
            if !ok {
                r.no(s, "only a call can give several values at once")
            }
            vals = r.eval(st, r.pure(c, func(name string) vkind {
                if b := r.binding(st, name); b != nil {
                    return b.kind
                }
                return kNone
            }), nil, c)
        } else {
            for _, x := range s.Results {
                vals = append(vals, r.value(st, x))
            }
        }
        if len(vals) != len(r.f.results) {
            r.no(s, "this return does not match the results")
        }
        // The results may be shadowed here; a return assigns the real ones.
        for i, p := range r.f.results {
            st.vars[p.name] = &binding{kind: p.kind, hist: []*expr{vals[i]}}
        }
    }
    r.leave(st, s.Pos())
}

func (r *run) leave(st *state, at token.Pos) {
    for _, a := range r.f.ensures {
        if a.e != nil {
            r.check(st, r.formula(st, a, true), a, nil, point(at), "Ensures is not proved for the return at {ref}: "+a.text)
        }
    }
    for _, a := range r.f.exsures {
        if a.e != nil {
            r.check(st, not(r.formula(st, a, true)), a, nil, point(at), "Exsures is not ruled out for the return at {ref}: "+a.text)
        }
    }
}

Loops

A loop can’t be walked, because nobody knows how many times it goes round. The invariant is what cuts it down to size:

  1. The invariant is proved for the state that reaches the loop.
  2. Every variable the loop assigns to is forgotten. It gets a fresh symbol, and all that is assumed about the new symbols is the invariant (plus whatever the enclosing blocks promised to preserve). This state stands for the start of any iteration.
  3. From there, with the condition true, the body is walked once. At its end the invariant is proved again, and the variant is proved to have been non-negative and to have gotten smaller.
  4. From the same state, with the condition false, the walk carries on after the loop.

A break leaves with whatever is known at that point, which may not include the invariant. A range loop is a counted loop whose counter can’t be tampered with, so it needs no variant to be known to end.

// loop is a for statement, or a range statement read as one.
type loop struct {
    stmt ast.Stmt
    init ast.Stmt
    cond ast.Expr
    post ast.Stmt
    body *ast.BlockStmt
    over *ast.RangeStmt
}

// loop cuts a loop at its invariant. The invariant is proved on entry;
// then everything the loop assigns is forgotten, the invariant is assumed,
// and one iteration has to give it back, with a smaller variant. What
// follows the loop knows the invariant and that the condition is false.
func (r *run) loop(st *state, l *loop, pending []*annot) outcomes {
    var invs, vars []*annot
    notes := append(append([]*annot{}, pending...), r.trailing(l.body)...)
    for _, a := range notes {
        a.used = true
        if a.e == nil {
            continue
        }
        if a.kind == "Invariant" {
            invs = append(invs, a)
        } else {
            vars = append(vars, a)
        }
    }
    if len(vars) == 0 && l.over == nil {
        r.v.report(r.v.position(l.stmt.Pos()), r.v.position(l.stmt.Pos()).Col+3, true, "this loop has no Variant, so nothing shows that it terminates")
    }
    st.push()
    depth := len(st.scopes)
    var out outcomes
    starts := []*state{st}
    var index string // the counter of a range loop
    var bound *expr
    if l.over != nil {
        index, bound = r.rangeStart(st, l.over)
    } else if l.init != nil {
        starts = r.stmt(st, l.init, nil).normal
    }
    for _, s0 := range starts {
        for _, a := range invs {
            r.check(s0, r.formula(s0, a, false), a, nil, l.stmt, "Invariant does not hold when the loop at {ref} is entered: "+a.text)
        }
        head := s0.clone()
        for _, name := range r.assigned(l) {
            if b := head.vars[name]; b != nil && (b.kind == kInt || b.kind == kBool) {
                b.hist = []*expr{r.fresh(name, b.kind)}
            }
        }
        if index != "" {
            b := head.vars[index]
            b.hist = []*expr{r.fresh(index, kInt)}
            r.assume(head, mk(opAnd, mk(opLe, num(0), b.hist[0]), mk(opLe, b.hist[0], bound)))
        }
        for _, a := range invs {
            r.assume(head, r.formula(head, a, false))
        }
        for _, k := range head.preserved {
            r.assume(head, r.formula(head, k.a, false))
        }
        var cond *expr
        switch {
        case index != "":
            cond = mk(opLt, head.vars[index].hist[0], bound)
        case l.cond != nil:
            cond = r.value(head, l.cond)
        default:
            cond = eTrue
        }
        exit := head.clone()
        r.assume(exit, not(cond))
        if r.feasible(exit) {
            out.normal = append(out.normal, exit)
        }
        body := head
        r.assume(body, cond)
        if !r.feasible(body) {
            continue
        }
        var before []*expr
        for _, a := range vars {
            m := r.formula(body, a, false)
            before = append(before, m)
            r.check(body, mk(opLe, num(0), m), a, nil, nil, "Variant can be negative while the loop still runs: "+a.text)
        }
        if l.over != nil && l.over.Value != nil {
            if id, ok := l.over.Value.(*ast.Ident); ok && id.Name != "_" {
                body.push()
                arr := r.value(body, l.over.X)
                body.declare(id.Name, &binding{kind: kInt, hist: []*expr{mk(opSel, arr, body.vars[index].hist[0])}})
            }
        }
        o := r.block(body, l.body)
        out.normal = append(out.normal, o.brk...)
        for _, s := range append(o.normal, o.cont...) {
            s.popTo(depth)
            ends := []*state{s}
            if index != "" {
                b := s.vars[index]
                b.hist = append(b.hist, mk(opAdd, b.hist[len(b.hist)-1], num(1)))
            } else if l.post != nil {
                ends = r.stmt(s, l.post, nil).normal
            }
            for _, s := range ends {
                for _, a := range invs {
                    r.check(s, r.formula(s, a, false), a, nil, l.stmt, "Invariant is not maintained by the body of the loop at {ref}: "+a.text)
                }
                for i, a := range vars {
                    r.check(s, mk(opLt, r.formula(s, a, false), before[i]), a, nil, l.stmt, "Variant is not shown to decrease in the loop at {ref}: "+a.text)
                }
            }
        }
    }
    for _, s := range out.normal {
        s.popTo(depth - 1)
    }
    return out
}
// trailing finds the Invariant and Variant lines written inside a loop's
// body rather than above the loop: the ones no inner loop follows.
func (r *run) trailing(body *ast.BlockStmt) []*annot {
    var out []*annot
    for _, a := range r.blocks[body] {
        if a.kind != "Invariant" && a.kind != "Variant" {
            continue
        }
        var next ast.Stmt
        for _, s := range body.List {
            if s.Pos() > a.pos {
                next = s
                break
            }
        }
        if !isLoop(next) {
            out = append(out, a)
        }
    }
    return out
}

// rangeStart declares the counter of a range loop, under the name the
// program gives it if it gives it one, and finds where the counting stops.
func (r *run) rangeStart(st *state, s *ast.RangeStmt) (string, *expr) {
    if s.Tok == token.ASSIGN {
        r.no(s, "a range loop has to declare its variables with :=")
    }
    index := "range counter"
    if id, ok := s.Key.(*ast.Ident); ok && id.Name != "_" {
        index = id.Name
    }
    for _, name := range r.assigned(&loop{body: s.Body}) {
        if name == index {
            r.no(s, "the body assigns to the loop's own counter")
        }
    }
    var bound *expr
    if id, ok := s.X.(*ast.Ident); ok {
        if b := r.binding(st, id.Name); b != nil && b.kind == kSlice {
            bound = mk(opLen, b.hist[0])
        }
    }
    if sel, ok := s.X.(*ast.SelectorExpr); ok && bound == nil {
        if x, ok := sel.X.(*ast.Ident); ok {
            if b := r.binding(st, x.Name+"."+sel.Sel.Name); b != nil && b.kind == kSlice {
                bound = mk(opLen, b.hist[0])
            }
        }
    }
    if bound == nil {
        if s.Value != nil {
            r.no(s, "only a slice of integers or an integer can be ranged over")
        }
        bound = r.value(st, s.X)
        if isBool(bound) || bound.sort == sArr {
            r.no(s, "only a slice of integers or an integer can be ranged over")
        }
    }
    st.declare(index, &binding{kind: kInt, hist: []*expr{num(0)}})
    return index, bound
}

// assigned lists the variables a loop can change.
func (r *run) assigned(l *loop) []string {
    seen := map[string]bool{}
    visit := func(n ast.Node) bool {
        switch s := n.(type) {
        case *ast.AssignStmt:
            for _, x := range s.Lhs {
                if id, ok := x.(*ast.Ident); ok {
                    seen[id.Name] = true
                }
            }
        case *ast.IncDecStmt:
            if id, ok := s.X.(*ast.Ident); ok {
                seen[id.Name] = true
            }
        case *ast.RangeStmt:
            for _, x := range []ast.Expr{s.Key, s.Value} {
                if id, ok := x.(*ast.Ident); ok {
                    seen[id.Name] = true
                }
            }
        }
        return true
    }
    ast.Inspect(l.body, visit)
    if l.post != nil {
        ast.Inspect(l.post, visit)
    }
    out := make([]string, 0, len(seen))
    for n := range seen {
        out = append(out, n)
    }
    sort.Strings(out)
    return out
}

The prover

internal/vego/prove.go

An obligation is a list of hypotheses H and a goal G. A model is a choice of values for every variable that makes a formula true, and H \models G (H entails G) when every model of H is a model of G. The prover checks this by refutation. It hunts for a model of F = H \land \lnot G, a way for the hypotheses to hold and the goal to fail. If there is none, the goal follows. It gets there in four steps. Each one turns F into some F' and keeps one promise: if F has a model, so does F'. A step may drop information, which only makes F' harder to refute, but it may never add a constraint that some model of F breaks.

  1. Negations are pushed inward using De Morgan and \lnot\forall x.P \equiv \exists x.\lnot P, so the negated goal’s Forall becomes an Exists. These are equivalences, so the models are unchanged. Every Exists not under a Forall then gets a fresh constant as its witness: \exists x.P(x) becomes P(c) (Skolemization). A model of the first gives one of the second by letting c be the x it has. The constant must be fresh so nothing else constrains it. Under a Forall, the witness would depend on the bound variable and would have to be a function.
  2. Every Forall is replaced by a handful of its instances, at the terms that index a slice somewhere in the obligation and at the witnesses: \forall x.P(x) becomes P(t_1) \land \dots \land P(t_n). Each instance follows from the universal, so this only ever weakens it. The choice of terms decides whether a proof is found, never whether it’s right. If that isn’t enough, it’s done once more, at the terms the instances brought along, which is what a proof about A[Parent(k)] needs and what most proofs are better off without.
  3. What’s left has no quantifiers. Its atoms become linear constraints over polynomials. A product like x*y becomes an unknown of its own, its link to x and y forgotten (weakening again: give it the product’s value). Over the integers p < 0 is exactly p + 1 <= 0, so there are no strict inequalities.
  4. A search splits on the disjunctions. A \lor B has a model only if one side does, so every branch has to close. Fourier-Motzkin elimination closes a branch by pairing each lower bound l \le x with each upper bound x \le u to derive l \le u. Every derived constraint is a nonnegative combination of earlier ones, so it holds wherever they do, and reaching 1 \le 0 means the branch has no model. No rational solution means no integer one either, and the gcd rounding below only removes solutions that aren’t integers.

The promise chains. If F had a model, some branch at the end would too, so when every branch closes, the goal holds. It’s also why the shortcuts are safe. Too few instances, a forgotten product or a budget that runs out can leave a branch open, which is only “not proved”. None of them can close a branch that has a model.

Arithmetic is normalised into polynomials. That’s how (q+1)*d + (r-d) and q*d + r turn out to be the same thing without any search. They’re the same polynomial.

// A poly is a sum of monomials with integer coefficients. A monomial is a
// product of atoms, written as their names joined by sep; the constant's
// monomial is the empty string.
type poly map[string]int64

const sep = "\x1f"

func constant(v int64) poly {
    if v == 0 {
        return poly{}
    }
    return poly{"": v}
}

func (p poly) add(q poly, k int64) poly {
    out := poly{}
    for m, c := range p {
        out[m] = c
    }
    for m, c := range q {
        out[m] += k * c
        if out[m] == 0 {
            delete(out, m)
        }
    }
    return out
}

func (p poly) mul(q poly) poly {
    out := poly{}
    for m1, c1 := range p {
        for m2, c2 := range q {
            var atoms []string
            if m1 != "" {
                atoms = append(atoms, strings.Split(m1, sep)...)
            }
            if m2 != "" {
                atoms = append(atoms, strings.Split(m2, sep)...)
            }
            sort.Strings(atoms)
            m := strings.Join(atoms, sep)
            out[m] += c1 * c2
            if out[m] == 0 {
                delete(out, m)
            }
        }
    }
    return out
}

func (p poly) isConst() (int64, bool) {
    if len(p) == 0 {
        return 0, true
    }
    if c, ok := p[""]; ok && len(p) == 1 {
        return c, true
    }
    return 0, false
}

func (p poly) String() string {
    keys := make([]string, 0, len(p))
    for m := range p {
        keys = append(keys, m)
    }
    sort.Strings(keys)
    var b strings.Builder
    for _, m := range keys {
        fmt.Fprintf(&b, "%+d", p[m])
        if m != "" {
            b.WriteString("*" + strings.ReplaceAll(m, sep, "*"))
        }
    }
    if b.Len() == 0 {
        return "0"
    }
    return b.String()
}

Over the integers p < 0 is p + 1 <= 0, so there are no strict inequalities, and p <> 0 is a choice between p < 0 and p > 0.

// A node is a formula whose leaves are constraints: p <= 0, p = 0, or a
// boolean atom with a sign.
type node struct {
    kind  byte // 'a' and, 'o' or, 'l' p <= 0, 'e' p = 0, 'b' boolean atom, 't' true, 'f' false
    kids  []*node
    p     poly
    atom  string
    truth bool
}

var (
    nTrue  = &node{kind: 't'}
    nFalse = &node{kind: 'f'}
)

func le0(p poly) *node {
    if c, ok := p.isConst(); ok {
        if c <= 0 {
            return nTrue
        }
        return nFalse
    }
    return &node{kind: 'l', p: p}
}

func eq0(p poly) *node {
    if c, ok := p.isConst(); ok {
        if c == 0 {
            return nTrue
        }
        return nFalse
    }
    return &node{kind: 'e', p: p}
}

// lt0 is p < 0, which over the integers is p + 1 <= 0.
func lt0(p poly) *node { return le0(p.add(constant(1), 1)) }

func ne0(p poly) *node { return nOr(lt0(p), lt0(poly{}.add(p, -1))) }

func nAnd(kids ...*node) *node {
    var out []*node
    for _, k := range kids {
        switch k.kind {
        case 't':
        case 'f':
            return nFalse
        case 'a':
            out = append(out, k.kids...)
        default:
            out = append(out, k)
        }
    }
    if len(out) == 0 {
        return nTrue
    }
    if len(out) == 1 {
        return out[0]
    }
    return &node{kind: 'a', kids: out}
}

func nOr(kids ...*node) *node {
    var out []*node
    for _, k := range kids {
        switch k.kind {
        case 'f':
        case 't':
            return nTrue
        case 'o':
            out = append(out, k.kids...)
        default:
            out = append(out, k)
        }
    }
    if len(out) == 0 {
        return nFalse
    }
    if len(out) == 1 {
        return out[0]
    }
    return &node{kind: 'o', kids: out}
}
// The budgets keep a hopeless proof from holding up the editor. Running out
// of budget means "not proved", never "proved".
const (
    maxSteps       = 2000
    maxConstraints = 1500
    maxInstances   = 4000
    maxTerms       = 14
)

type prover struct {
    fresh     int
    steps     int
    instances int
    gaveUp    bool
    atoms     map[string]bool // atoms whose axioms have been stated
    axioms    []*node
    sels      map[string][]selAtom // by array
}

type selAtom struct {
    name string
    idx  poly
}

// proves reports whether the hypotheses entail the goal. It answers by
// refutation: the hypotheses and the goal's negation must have no model.
func proves(hyps []*expr, goal *expr) bool {
    // Most proofs need every Forall only at the terms that are already there.
    // Some need it again at the terms those instances mention, and that costs
    // much more, so it is the second thing to try and not the first.
    return attempt(hyps, goal, 1) || attempt(hyps, goal, 2)
}

func attempt(hyps []*expr, goal *expr, rounds int) bool {
    pr := &prover{atoms: map[string]bool{}, sels: map[string][]selAtom{}}
    f := nnf(and(conj(hyps), not(goal)), true)
    f = pr.skolem(f)
    for round := 0; round < rounds; round++ {
        f = pr.instantiate(f, pr.terms(f), round < rounds-1)
    }
    f = pr.instantiate(f, nil, false)
    g := pr.ground(f)
    pr.congruence()
    all := nAnd(append([]*node{g}, pr.axioms...)...)
    return pr.refute(nil, map[string]bool{}, []*node{all}) && !pr.gaveUp
}
// nnf pushes negations down to the atoms. pos is false under a negation.
func nnf(e *expr, pos bool) *expr {
    switch e.op {
    case opTrue, opFalse:
        if pos == (e.op == opTrue) {
            return eTrue
        }
        return eFalse
    case opNot:
        return nnf(e.args[0], !pos)
    case opAnd, opOr:
        o := e.op
        if !pos {
            o = opAnd + opOr - o
        }
        return mk(o, nnf(e.args[0], pos), nnf(e.args[1], pos))
    case opImp:
        if pos {
            return mk(opOr, nnf(e.args[0], false), nnf(e.args[1], true))
        }
        return mk(opAnd, nnf(e.args[0], true), nnf(e.args[1], false))
    case opIff:
        a, b := e.args[0], e.args[1]
        if pos {
            return mk(opAnd, mk(opOr, nnf(a, false), nnf(b, true)), mk(opOr, nnf(b, false), nnf(a, true)))
        }
        return mk(opOr, mk(opAnd, nnf(a, true), nnf(b, false)), mk(opAnd, nnf(b, true), nnf(a, false)))
    case opForall, opExists:
        o := e.op
        if !pos {
            o = opForall + opExists - o
        }
        return &expr{op: o, name: e.name, args: []*expr{nnf(e.args[0], pos)}}
    }
    if pos {
        return e
    }
    switch e.op {
    case opLt:
        return mk(opLe, e.args[1], e.args[0])
    case opLe:
        return mk(opLt, e.args[1], e.args[0])
    case opEq:
        return mk(opNe, e.args[0], e.args[1])
    case opNe:
        return mk(opEq, e.args[0], e.args[1])
    }
    return not(e) // a boolean atom
}

A witness can only be named once nothing universal sits above it. In Forall i : Exists j : P(i, j) the j depends on the i, so the Exists waits until the Forall has been instantiated, then gets a new witness per instance.

// skolem names a witness for every Exists that is not under a Forall. One
// under a Forall waits until the Forall has been instantiated.
func (pr *prover) skolem(e *expr) *expr {
    switch e.op {
    case opAnd, opOr:
        return mk(e.op, pr.skolem(e.args[0]), pr.skolem(e.args[1]))
    case opExists:
        pr.fresh++
        w := sym(fmt.Sprintf("%s?%d", e.name, pr.fresh), sInt)
        return pr.skolem(subst(e.args[0], e.name, w))
    }
    return e
}

// terms collects what a Forall might be worth instantiating with: the ground
// terms that index a slice, and the witnesses.
func (pr *prover) terms(e *expr) []*expr {
    seen := map[string]bool{}
    var out []*expr
    var walk func(e *expr, bound []string)
    walk = func(e *expr, bound []string) {
        if e == nil {
            return
        }
        if e.op == opForall || e.op == opExists {
            bound = append(bound[:len(bound):len(bound)], e.name)
        }
        candidate := e.op == opSel && len(e.args) == 2
        var t *expr
        if candidate {
            t = e.args[1]
        } else if e.op == opVar && strings.Contains(e.name, "?") {
            t, candidate = e, true
        }
        if candidate {
            free := true
            for _, n := range names(t) {
                for _, b := range bound {
                    free = free && n != b
                }
            }
            if key := t.String(); free && !seen[key] && len(out) < maxTerms {
                seen[key] = true
                out = append(out, t)
            }
        }
        for _, a := range e.args {
            walk(a, bound)
        }
    }
    walk(e, nil)
    return out
}

// instantiate replaces every outermost Forall by its instances at the terms,
// and keeps the Forall itself for a later round if asked to.
func (pr *prover) instantiate(e *expr, terms []*expr, keep bool) *expr {
    switch e.op {
    case opAnd, opOr:
        return mk(e.op, pr.instantiate(e.args[0], terms, keep), pr.instantiate(e.args[1], terms, keep))
    case opForall:
        out := eTrue
        if keep {
            out = e
        }
        for _, t := range terms {
            if pr.instances++; pr.instances > maxInstances {
                pr.gaveUp = true
                break
            }
            inst := pr.skolem(subst(e.args[0], e.name, t))
            out = and(out, pr.instantiate(inst, terms, keep))
        }
        return out
    }
    return e
}
// ground turns a quantifier-free formula into constraints.
func (pr *prover) ground(e *expr) *node {
    switch e.op {
    case opTrue:
        return nTrue
    case opFalse:
        return nFalse
    case opAnd:
        return nAnd(pr.ground(e.args[0]), pr.ground(e.args[1]))
    case opOr:
        return nOr(pr.ground(e.args[0]), pr.ground(e.args[1]))
    case opForall:
        return nTrue // a fact we could not use
    case opExists:
        pr.gaveUp = true
        return nTrue
    case opNot:
        return &node{kind: 'b', atom: e.args[0].String(), truth: false}
    case opVar, opCall:
        return &node{kind: 'b', atom: e.String(), truth: true}
    }
    if len(e.args) != 2 || isBool(e.args[0]) || isBool(e.args[1]) {
        pr.gaveUp = true // a number where a formula belongs, or the other way round
        return nTrue
    }
    d := pr.poly(e.args[0]).add(pr.poly(e.args[1]), -1)
    switch e.op {
    case opLe:
        return le0(d)
    case opLt:
        return lt0(d)
    case opEq:
        return eq0(d)
    case opNe:
        return ne0(d)
    }
    pr.gaveUp = true
    return nTrue
}

Anything that isn’t addition or multiplication by a constant becomes an atom, an unknown of its own. The facts that make it more than an unknown get added as axioms the first time it shows up: a length is never negative, a quotient and its remainder add up, equal indices select equal elements, and a product of non-negative numbers is non-negative.

// poly normalises an integer term. Whatever is not arithmetic becomes an
// atom: a variable, an element of a slice, a length, a quotient.
func (pr *prover) poly(e *expr) poly {
    switch e.op {
    case opNum:
        return constant(e.val)
    case opVar:
        return poly{e.name: 1}
    case opAdd:
        return pr.poly(e.args[0]).add(pr.poly(e.args[1]), 1)
    case opSub:
        return pr.poly(e.args[0]).add(pr.poly(e.args[1]), -1)
    case opNeg:
        return poly{}.add(pr.poly(e.args[0]), -1)
    case opMul:
        p := pr.poly(e.args[0]).mul(pr.poly(e.args[1]))
        pr.products(p)
        return p
    case opLen:
        name := "len(" + e.args[0].name + ")"
        if !pr.atoms[name] {
            pr.atoms[name] = true
            pr.axioms = append(pr.axioms, le0(poly{name: -1}))
        }
        return poly{name: 1}
    case opSel:
        idx := pr.poly(e.args[1])
        arr := e.args[0].name
        name := arr + "[" + idx.String() + "]"
        if !pr.atoms[name] {
            pr.atoms[name] = true
            pr.sels[arr] = append(pr.sels[arr], selAtom{name, idx})
        }
        return poly{name: 1}
    case opDiv, opMod:
        return pr.divide(e)
    }
    pr.gaveUp = true
    return poly{e.String(): 1}
}

// divide states what Go's division means. For a = q*b + r with b > 0, the
// remainder has the sign of a and is smaller than b.
func (pr *prover) divide(e *expr) poly {
    a, b := pr.poly(e.args[0]), pr.poly(e.args[1])
    key := "(" + a.String() + ")/(" + b.String() + ")"
    q, r := "quo"+key, "rem"+key
    if !pr.atoms[q] {
        pr.atoms[q] = true
        qb := poly{q: 1}.mul(b)
        pr.products(qb)
        // a = q*b + r
        pr.axioms = append(pr.axioms, eq0(a.add(qb, -1).add(poly{r: 1}, -1)))
        pos := lt0(poly{}.add(b, -1))                    // 0 < b
        nonneg, nonpos := le0(poly{}.add(a, -1)), le0(a) // 0 <= a, a <= 0
        small := lt0(poly{r: 1}.add(b, -1))              // r < b
        smallNeg := lt0(poly{r: -1}.add(b, -1))          // -r < b
        pr.axioms = append(pr.axioms,
            nOr(negate(pos), negate(nonneg), nAnd(le0(poly{r: -1}), small, le0(poly{q: -1}))),
            nOr(negate(pos), negate(nonpos), nAnd(le0(poly{r: 1}), smallNeg, le0(poly{q: 1}))))
    }
    if e.op == opDiv {
        return poly{q: 1}
    }
    return poly{r: 1}
}

// negate works on the constraints divide builds: p <= 0 becomes -p + 1 <= 0.
func negate(n *node) *node {
    switch n.kind {
    case 't':
        return nFalse
    case 'f':
        return nTrue
    case 'l':
        return lt0(poly{}.add(n.p, -1))
    }
    panic("negate: not an inequality")
}

// products says the little that is known about x*y: it is not negative when
// neither factor is, and a square never is.
func (pr *prover) products(p poly) {
    for m := range p {
        atoms := strings.Split(m, sep)
        if len(atoms) != 2 || pr.atoms[m] {
            continue
        }
        pr.atoms[m] = true
        x, y := atoms[0], atoms[1]
        if x == y {
            pr.axioms = append(pr.axioms, le0(poly{m: -1}))
            continue
        }
        pr.axioms = append(pr.axioms, nOr(lt0(poly{x: 1}), lt0(poly{y: 1}), le0(poly{m: -1})),
            nOr(lt0(poly{x: 1}), le0(poly{y: 1}), le0(poly{x: 1, m: -1})), // 0 <= x and 1 <= y: x <= xy
            nOr(lt0(poly{y: 1}), le0(poly{x: 1}), le0(poly{y: 1, m: -1})))
    }
}

// congruence says that equal indices select equal elements.
func (pr *prover) congruence() {
    pairs := 0
    for _, list := range pr.sels {
        for i := range list {
            for j := i + 1; j < len(list); j++ {
                d := list[i].idx.add(list[j].idx, -1)
                if c, ok := d.isConst(); ok && c != 0 {
                    continue
                }
                if pairs++; pairs > 80 {
                    return
                }
                pr.axioms = append(pr.axioms, nOr(ne0(d), eq0(poly{list[i].name: 1, list[j].name: -1})))
            }
        }
    }
}

The search keeps the constraints of the branch it’s in and the disjunctions it hasn’t decided yet. Before it splits, it drops every disjunct that already contradicts the branch. A disjunction with one disjunct left is a fact, and one with none left closes the branch. That’s unit propagation, and it’s what stops the instances of a quantifier from multiplying the work.

// refute reports whether the constraints, the boolean assignment and the
// formulas still to do are unsatisfiable together.
func (pr *prover) refute(cons []poly2, bools map[string]bool, todo []*node) bool {
    if pr.steps++; pr.steps > maxSteps {
        pr.gaveUp = true
        return false
    }
    cons = cons[:len(cons):len(cons)]
    var ors []*node
    copied := false
    for len(todo) > 0 {
        n := todo[len(todo)-1]
        todo = todo[: len(todo)-1 : len(todo)-1]
        switch n.kind {
        case 'f':
            return true
        case 'a':
            todo = append(todo, n.kids...)
        case 'o':
            ors = append(ors, n)
        case 'l', 'e':
            cons = append(cons, poly2{n.p, n.kind == 'e'})
        case 'b':
            if v, ok := bools[n.atom]; ok && v != n.truth {
                return true
            }
            if !copied {
                c := make(map[string]bool, len(bools)+1)
                for k, v := range bools {
                    c[k] = v
                }
                bools, copied = c, true
            }
            bools[n.atom] = n.truth
        }
    }
    if pr.unsat(cons) {
        return true
    }
    if len(ors) == 0 {
        return false
    }
    // Drop the disjuncts that are already impossible; a disjunction with one
    // disjunct left is a fact, and one with none closes the branch.
    var rest []*node
    var units []*node
    for _, o := range ors {
        var live []*node
        for _, k := range o.kids {
            if !pr.dead(cons, bools, k) {
                live = append(live, k)
            }
        }
        switch len(live) {
        case 0:
            return true
        case 1:
            units = append(units, live[0])
        default:
            rest = append(rest, &node{kind: 'o', kids: live})
        }
    }
    if len(units) > 0 {
        return pr.refute(cons, bools, append(rest, units...))
    }
    sort.SliceStable(rest, func(i, j int) bool { return len(rest[i].kids) < len(rest[j].kids) })
    for _, k := range rest[0].kids {
        if !pr.refute(cons, bools, append(rest[1:len(rest):len(rest)], k)) {
            return false
        }
    }
    return true
}

// dead reports whether a disjunct is a constraint that contradicts what is
// already known. Anything more complicated is left for the branch to find.
func (pr *prover) dead(cons []poly2, bools map[string]bool, n *node) bool {
    switch n.kind {
    case 'f':
        return true
    case 'b':
        v, ok := bools[n.atom]
        return ok && v != n.truth
    case 'l', 'e':
        return pr.unsat(append(cons[:len(cons):len(cons)], poly2{n.p, n.kind == 'e'}))
    }
    return false
}

// poly2 is one constraint: p <= 0, or p = 0.
type poly2 struct {
    p  poly
    eq bool
}

Fourier-Motzkin elimination

Okay, how do you decide whether a pile of linear inequalities can all hold at once? Pick a variable. Pair every constraint that bounds it from above with every one that bounds it from below, and keep the sums, in which the variable cancels out. Repeat until no variables are left. The constraints are contradictory exactly when some c \le 0 with a positive c remains. That’s complete over the rationals. Over the integers it’s only sound, and two small tricks make it sharper: equalities are solved and substituted first, and every constraint is divided by the gcd of its coefficients with the constant rounded, so 2x \le 1 becomes x \le 0.

// unsat decides whether linear constraints have no rational solution, with
// the rounding that makes it sharper over the integers. Every monomial is a
// variable here, so a product is just another unknown.
func (pr *prover) unsat(cons []poly2) bool {
    var eqs, les []poly
    for _, c := range cons {
        if c.eq {
            eqs = append(eqs, c.p)
        } else {
            les = append(les, c.p)
        }
    }
    eqs, les = squeeze(eqs, les)
    eqs, les = unfoldProducts(eqs, les)
    // Equalities first: solve each for one variable and substitute.
    for len(eqs) > 0 {
        e := eqs[0]
        eqs = eqs[1:]
        e, ok := tighten(e, true)
        if !ok {
            return true
        }
        if len(e) == 0 {
            continue
        }
        v := pivot(e)
        for i := range eqs {
            eqs[i] = eliminate(eqs[i], e, v)
        }
        for i := range les {
            les[i] = eliminate(les[i], e, v)
        }
    }
    for {
        // Normalise, drop what is trivially true, and keep the tightest of the
        // constraints that differ only in their constant.
        best := map[string]poly{}
        for _, p := range les {
            p, _ := tighten(p, false)
            if c, ok := p.isConst(); ok {
                if c > 0 {
                    return true
                }
                continue
            }
            k := key(p)
            if old, ok := best[k]; !ok || p[""] > old[""] {
                best[k] = p
            }
        }
        if len(best) == 0 {
            return false
        }
        if len(best) > maxConstraints {
            pr.gaveUp = true
            return false
        }
        les = les[:0]
        keys := make([]string, 0, len(best))
        for k := range best {
            keys = append(keys, k)
        }
        sort.Strings(keys)
        for _, k := range keys {
            les = append(les, best[k])
        }
        // Eliminate the variable that multiplies the fewest pairs.
        up, down := map[string]int{}, map[string]int{}
        for _, p := range les {
            for m, c := range p {
                if m == "" {
                    continue
                }
                if c > 0 {
                    up[m]++
                } else {
                    down[m]++
                }
            }
        }
        v, cost := "", -1
        for _, p := range les {
            for m := range p {
                if m == "" {
                    continue
                }
                if c := up[m] * down[m]; cost < 0 || c < cost || c == cost && m < v {
                    v, cost = m, c
                }
            }
        }
        var next, pos, neg []poly
        for _, p := range les {
            switch c := p[v]; {
            case c > 0:
                pos = append(pos, p)
            case c < 0:
                neg = append(neg, p)
            default:
                next = append(next, p)
            }
        }
        for _, p := range pos {
            for _, n := range neg {
                a, b := p[v], -n[v]
                if big(a) || big(b) {
                    pr.gaveUp = true
                    return false
                }
                c := poly{}.add(p, b).add(n, a)
                for _, k := range c {
                    if big(k) {
                        pr.gaveUp = true
                        return false
                    }
                }
                next = append(next, c)
            }
        }
        les = next
    }
}
// squeeze finds the equalities hidden in pairs of inequalities: p <= 0 and
// -p <= 0.
func squeeze(eqs, les []poly) ([]poly, []poly) {
    byKey := map[string]bool{}
    for _, p := range les {
        byKey[p.String()] = true
    }
    for _, p := range les {
        if len(p) > 0 && byKey[poly{}.add(p, -1).String()] {
            eqs = append(eqs, p)
        }
    }
    return eqs, les
}

// unfoldProducts uses an equality x = e inside the products x occurs in.
// Elimination cannot do that, because to it x*y is one unknown and x another;
// but i = n + 1 is exactly what turns i*i into something about n.
func unfoldProducts(eqs, les []poly) ([]poly, []poly) {
    for round := 0; round < 8; round++ {
        x, by := "", poly(nil)
        for _, e := range eqs {
            for m, c := range e {
                if m == "" || strings.Contains(m, sep) || c != 1 && c != -1 || !inProduct(m, eqs, les) {
                    continue
                }
                rest := poly{}.add(e, -c) // x = rest, once x's own term is taken out
                delete(rest, m)
                if mentions(rest, m) {
                    continue
                }
                if x == "" || m < x {
                    x, by = m, rest
                }
            }
        }
        if x == "" {
            break
        }
        for i := range eqs {
            eqs[i] = substitute(eqs[i], x, by)
        }
        for i := range les {
            les[i] = substitute(les[i], x, by)
        }
        eqs = append(eqs, poly{x: 1}.add(by, -1)) // what substituting made trivial
    }
    return eqs, les
}

func mentions(p poly, x string) bool {
    for m := range p {
        for _, a := range strings.Split(m, sep) {
            if a == x {
                return true
            }
        }
    }
    return false
}

func inProduct(x string, lists ...[]poly) bool {
    for _, list := range lists {
        for _, p := range list {
            for m := range p {
                if strings.Contains(m, sep) && mentions(poly{m: 1}, x) {
                    return true
                }
            }
        }
    }
    return false
}

func substitute(p poly, x string, by poly) poly {
    if !mentions(p, x) {
        return p
    }
    out := poly{}
    for m, c := range p {
        term := poly{"": c}
        for _, a := range strings.Split(m, sep) {
            switch {
            case a == x:
                term = term.mul(by)
            case a != "":
                term = term.mul(poly{a: 1})
            }
        }
        out = out.add(term, 1)
    }
    return out
}
func big(v int64) bool { return v > 1<<30 || v < -(1<<30) }

func key(p poly) string {
    ms := make([]string, 0, len(p))
    for m := range p {
        if m != "" {
            ms = append(ms, m)
        }
    }
    sort.Strings(ms)
    var b strings.Builder
    for _, m := range ms {
        fmt.Fprintf(&b, "%d*%s;", p[m], m)
    }
    return b.String()
}

// tighten divides a constraint by the gcd of its coefficients. Over the
// integers the constant of an inequality rounds up, and an equality whose
// constant does not divide has no solution.
func tighten(p poly, eq bool) (poly, bool) {
    var g int64
    for m, c := range p {
        if m != "" {
            g = gcd(g, c)
        }
    }
    if g == 0 {
        c := p[""]
        return p, !eq || c == 0
    }
    if g == 1 {
        return p, true
    }
    out := poly{}
    for m, c := range p {
        if m != "" {
            out[m] = c / g
        }
    }
    c := p[""]
    if eq {
        if c%g != 0 {
            return p, false
        }
        c /= g
    } else {
        c = ceilDiv(c, g)
    }
    if c != 0 {
        out[""] = c
    }
    return out, true
}

func gcd(a, b int64) int64 {
    if a < 0 {
        a = -a
    }
    if b < 0 {
        b = -b
    }
    for b != 0 {
        a, b = b, a%b
    }
    return a
}

func ceilDiv(a, b int64) int64 {
    q := a / b
    if a%b != 0 && (a < 0) == (b < 0) {
        q++
    }
    return q
}

// pivot picks the variable of an equality to solve for: one with coefficient
// ±1 if there is one, so that substituting it loses nothing over the integers.
func pivot(e poly) string {
    v := ""
    for m, c := range e {
        if m == "" {
            continue
        }
        unit := c == 1 || c == -1
        bestUnit := v != "" && (e[v] == 1 || e[v] == -1)
        if v == "" || unit && !bestUnit || unit == bestUnit && m < v {
            v = m
        }
    }
    return v
}

// eliminate removes v from p using the equality e = 0.
func eliminate(p, e poly, v string) poly {
    c := p[v]
    if c == 0 {
        return p
    }
    a := e[v]
    // |a|*p - sign(a)*c*e has no v, and scaling an inequality by |a| > 0 is fine.
    if a < 0 {
        return poly{}.add(p, -a).add(e, c)
    }
    return poly{}.add(p, a).add(e, -c)
}

Tests

internal/vego/vego_test.go

A verifier can be wrong in two ways, and they are not equally bad. If it fails to prove something true, somebody strengthens an invariant. If it proves something false, it’s worse than useless. So every program here that is proved also gets broken, one small change at a time, and each broken version has to be rejected for the right reason.

func verifySrc(t *testing.T, src string) ([]Finding, Summary) {
    t.Helper()
    return Verify([]Source{{"t.go", []byte("package p\n" + src)}})
}

func expectProved(t *testing.T, src string, n int) {
    t.Helper()
    fs, sum := verifySrc(t, src)
    for _, f := range fs {
        if !f.Warning {
            t.Errorf("unexpected: %d:%d %s", f.Line+1, f.Col+1, f.Message)
        }
    }
    if sum.Proved != n {
        t.Errorf("proved %d of %d, want %d", sum.Proved, sum.Functions, n)
    }
}

func expectFail(t *testing.T, src, want string) {
    t.Helper()
    fs, _ := verifySrc(t, src)
    for _, f := range fs {
        if !f.Warning && strings.Contains(f.Message, want) {
            return
        }
    }
    t.Errorf("no error containing %q in %v", want, fs)
}

const divide = `
//@ Requires n >= 0 ^ d > 0
func Divide(n, d int) (q, r int) {
    r = n
    //@ Invariant n = q*d + r ^ r >= 0
    //@ Variant r
    for r >= d {
        q++
        r -= d
    }
    return q, r
}
//@ Ensures n = q*d + r
//@ Ensures 0 <= r < d
`

func TestDivide(t *testing.T) { expectProved(t, divide, 1) }

// gofmt moves annotations around: //@ becomes // @ in a doc comment, and an
// Ensures gets a blank line above it. The proof has to come out the same, and
// so does the refusal.
func TestSurvivesGofmt(t *testing.T) {
    for _, src := range []string{divide, search} {
        formatted, err := format.Source([]byte("package p\n" + src))
        if err != nil {
            t.Fatal(err)
        }
        text := strings.TrimPrefix(string(formatted), "package p\n")
        if !strings.Contains(text, "// @ Requires") || !strings.Contains(text, "}\n\n//@ Ensures") {
            t.Fatalf("gofmt no longer does what this test is about:\n%s", text)
        }
        expectProved(t, text, 1)
    }
    formatted, _ := format.Source([]byte("package p\n" + strings.Replace(divide, "r -= d", "r -= 1", 1)))
    expectFail(t, strings.TrimPrefix(string(formatted), "package p\n"), "Invariant is not maintained")
}

// A contract may also be written whole above its function.
func TestContractAbove(t *testing.T) {
    const above = `
func First() (x int) { return 1 }

//@ Requires n >= 0
//@ Ensures r = n + 1
func Next(n int) (r int) { return n + 1 }
`
    expectProved(t, above, 1)
    expectFail(t, strings.Replace(above, "return n + 1", "return n", 1), "Ensures is not proved")
}

// What the verifier cannot follow, it refuses.
func TestRefusals(t *testing.T) {
    expectFail(t, `
//@ Requires 0 <= i ^ i < |A|
func Set(A []int, i int) (x int) {
    A[i] = 0
    return A[i]
}
//@ Ensures x = 0
`, "only variables can be assigned to")
    expectFail(t, `
func scramble(A []int) {
    for i := range A {
        A[i] = 0
    }
}

//@ Requires |A| > 0 ^ A[0] = 1
func Keeps(A []int) (x int) {
    scramble(A)
    return A[0]
}
//@ Ensures x = 1
`, "scramble has no annotations")
    expectFail(t, `
//@ Requires n >= 0
func Bad(n int) (x int) {
    //@ Assert n + 1
    return n
}
`, "this has to be a formula")
    expectFail(t, `
//@ Requires n >= 0
func Bad(n int) (x int) {
    //@ Variant n > 0
    for n > 0 {
        n--
    }
    return n
}
`, "this has to be an integer expression")
    fs, sum := verifySrc(t, `
type Counter interface {
    //@ Requires self.Value >= 0
    Increment()
}

//@ Requires n >= 0
func Id(n int) (x int) { return n }
//@ Ensures x = n
`)
    if sum.Proved != 1 || len(fs) != 1 || !fs[0].Warning {
        t.Errorf("findings = %+v, sum = %+v", fs, sum)
    }
}

func TestDivideWrong(t *testing.T) {
    expectFail(t, strings.Replace(divide, "r -= d", "r -= 1", 1), "Invariant is not maintained")
    expectFail(t, strings.Replace(divide, "d > 0", "d >= 0", 1), "Variant is not shown to decrease")
    expectFail(t, strings.Replace(divide, "0 <= r < d", "0 <= r <= d - 2", 1), "Ensures is not proved")
}

func TestGauss(t *testing.T) {
    expectProved(t, `
//@ Requires n >= 0
func Sum(n int) (s int) {
    //@ Invariant 0 <= i <= n + 1 ^ 2*s = i*(i-1)
    //@ Variant n + 1 - i
    for i := 0; i <= n; i++ {
        s += i
    }
    return s
}
//@ Ensures 2*s = n*(n+1)
`, 1)
}

func TestSqrt(t *testing.T) {
    expectProved(t, `
//@ Requires n >= 0
func Isqrt(n int) (r int) {
    //@ Invariant r >= 0 ^ r*r <= n
    //@ Variant n - r*r
    for (r+1)*(r+1) <= n {
        r++
    }
    return r
}
//@ Ensures r*r <= n ^ n < (r+1)*(r+1)
`, 1)
}

const search = `
//@ Predicate Sorted(A) ::= Forall i in [0, |A|) : Forall j in [i, |A|) : A[i] <= A[j]

//@ Requires Sorted(A)
func Search(A []int, x int) (at int, found bool) {
    lo, hi := 0, len(A)
    //@ Invariant 0 <= lo <= hi <= |A|
    //@ Invariant A[0:lo) < x ^ x < A[hi:|A|)
    //@ Variant hi - lo
    for lo < hi {
        mid := lo + (hi-lo)/2
        if A[mid] == x {
            return mid, true
        } else if A[mid] < x {
            lo = mid + 1
        } else {
            hi = mid
        }
    }
    return 0, false
}
//@ Ensures found -> 0 <= at < |A| ^ A[at] = x
//@ Ensures ~found -> x <> A[:]
`

func TestSearch(t *testing.T) { expectProved(t, search, 1) }

func TestSearchWrong(t *testing.T) {
    expectFail(t, strings.Replace(search, "lo = mid + 1", "lo = mid", 1), "Variant is not shown to decrease")
    expectFail(t, strings.Replace(search, "hi = mid\n", "hi = mid - 1\n", 1), "Invariant is not maintained")
    expectFail(t, strings.Replace(search, "//@ Requires Sorted(A)\n", "", 1), "Invariant is not maintained")
}

func TestSafety(t *testing.T) {
    expectFail(t, `
//@ Requires i >= 0
func At(A []int, i int) (x int) { return A[i] }
`, "index may be out of range")
    expectFail(t, `
//@ Requires a >= 0
func Div(a, b int) (x int) { return a / b }
`, "divisor may be zero")
}

func TestCallsAndRecursion(t *testing.T) {
    expectProved(t, `
//@ Requires n >= 0
//@ Measure n
func Fact(n int) (f int) {
    //@ BaseCase n = 0
    if n == 0 {
        return 1
    }
    return n * Fact(n-1)
}
//@ Ensures f >= 1

//@ Requires a >= 0 ^ b >= 0
func Use(a, b int) (x int) {
    return Fact(a) + Fact(b)
}
//@ Ensures x >= 2
`, 2)
    expectFail(t, `
//@ Requires n >= 0
//@ Measure n
func F(n int) (f int) {
    if n == 0 {
        return 1
    }
    return F(n)
}
`, "Measure is not shown to get smaller")
}

func TestPrimesAndPreserves(t *testing.T) {
    expectProved(t, `
//@ Requires a = a
func P(a int) (x int) {
    x = 0
    x = x + 1
    x = x + 2
    //@ Assert x = 0 ^ x' = 1 ^ x'' = 3
    y := a
    //@ Preserves x + y >= a
    x++
    y--
    x++
    return x + y
}
//@ Ensures x >= a
`, 1)
}

// The examples are documentation, and documentation that claims a proof had
// better have one.
func TestExamples(t *testing.T) {
    for name, want := range map[string]int{"isqrt": 1, "vego-tutorial": 15, "heap": 10} {
        src, err := os.ReadFile("../../examples/" + name + ".lit.md")
        if err != nil {
            t.Skip(err)
        }
        diags, sum := Prove(lit.Parse(name+".lit.md", src).Tangle())
        for _, d := range diags {
            if d.Severity == lit.SevError {
                t.Errorf("%s:%d: %s", name, d.Line+1, d.Message)
            }
        }
        if sum.Proved != want || sum.Functions != want {
            t.Errorf("%s: proved %d of %d, want %d", name, sum.Proved, sum.Functions, want)
        }
    }
}

// A failure is reported where the annotation is in the Markdown, and the
// line it mentions is a line of the Markdown too.
// litgo proves the arithmetic of its own source map. The test names the two
// functions, so that deleting their annotations does not pass for a proof.
func TestLitgoProvesItself(t *testing.T) {
    src, err := os.ReadFile("../../litgo.lit.md")
    if err != nil {
        t.Skip(err)
    }
    diags, sum := Prove(lit.Parse("litgo.lit.md", src).Tangle())
    for _, d := range diags {
        if d.Severity == lit.SevError {
            t.Errorf("litgo.lit.md:%d: %s", d.Line+1, d.Message)
        }
    }
    proved := strings.Join(sum.Names, " ")
    if !strings.Contains(proved, "Seg.out") || !strings.Contains(proved, "Seg.src") || sum.Proved != sum.Functions {
        t.Errorf("proved %d of %d: %s", sum.Proved, sum.Functions, proved)
    }
}

func TestProveMapsToTheDocument(t *testing.T) {
    doc := "<!-- package: halve -->\n# Halving\n\n```go\nfunc Halve(n int) (h int) {\n\t// <<halve>>\n}\n//@ Ensures 2*h = n\n```\n\n" +
        "<!-- chunk: halve -->\n```go\nreturn n / 2\n```\n"
    diags, sum := Prove(lit.Parse("halve.lit.md", []byte(doc)).Tangle())
    if sum.Proved != 0 || len(diags) != 1 {
        t.Fatalf("diags = %+v, sum = %+v", diags, sum)
    }
    d := diags[0]
    if d.Line != 7 || d.Source != "vego" || !strings.Contains(d.Message, "the return at line 13") {
        t.Errorf("diag = %+v", d)
    }
}

func TestOneLine(t *testing.T) {
    expectProved(t, `
type Seg struct{ OutCol, Len, SrcLine, SrcCol int }

//@ Requires s.SrcCol <= col ^ col <= s.SrcCol + s.Len
func (s Seg) out(col int) (c int) { return s.OutCol + col - s.SrcCol }
//@ Ensures s.OutCol <= c ^ c <= s.OutCol + s.Len
//@ Ensures s.src(c) = col

//@ Requires s.OutCol <= col ^ col <= s.OutCol + s.Len
func (s Seg) src(col int) (c int) { return s.SrcCol + col - s.OutCol }
//@ Ensures s.SrcCol <= c ^ c <= s.SrcCol + s.Len
//@ Ensures s.out(c) = col
`, 2)
}

The command

main.go

Half of the commands are for people. The other half are the editor protocol: the document goes in on standard input, and JSON comes out. Why standard input? Because the buffer you are editing is rarely the same as the file on disk.

const usage = `litgo — literate Go

  litgo tangle [--force] [--stdout] FILE.lit.md...   write the Go source
  litgo run    [--vet] [--no-prove] [--timeout 30s] FILE [-- ARGS]
                                                     tangle, prove, compile and run
  litgo weave  [--to pdf|html|typ|md] [-o OUT] FILE  write a PDF through Typst, or a web page
  litgo check  FILE.lit.md...                        chunk, syntax, type and proof errors; writes nothing
  litgo prove  [--json] FILE.lit.md...               check the //@ annotations: contracts, invariants, termination
  litgo cat    FILE.lit.md                           print with math and diagrams rendered
  litgo fmt    [-w] FILE.lit.md                      gofmt the code blocks

Editor protocol (document on stdin, JSON on stdout):
  litgo analyze  [--path P] [--upright]
  litgo untangle --start L --end L [--start-col C --end-col C] [--name N]
  litgo rename   --from NAME --to NAME
  litgo run --json FILE                              one JSON event per line
  litgo lsp [--gopls PATH|off]                       language server on stdin/stdout: errors as you type,
                                                     and gopls for the Go in the code blocks

Positions in the editor protocol are 0-based; columns are bytes.
`

const VERSION = "0.2.3"

func main() {
    if len(os.Args) < 2 {
        fmt.Fprint(os.Stderr, usage)
        os.Exit(2)
    }
    cmd, args := os.Args[1], os.Args[2:]
    switch cmd {
    case "tangle":
        os.Exit(cmdTangle(args))
    case "run":
        os.Exit(cmdRun(args))
    case "weave":
        os.Exit(cmdWeave(args))
    case "check":
        os.Exit(cmdCheck(args))
    case "prove":
        os.Exit(cmdProve(args))
    case "cat":
        os.Exit(cmdCat(args))
    case "fmt":
        os.Exit(cmdFmt(args))
    case "analyze":
        os.Exit(cmdAnalyze(args))
    case "untangle":
        os.Exit(cmdUntangle(args))
  case "version":
    os.Exit(cmdVersion(args))
    case "rename":
        os.Exit(cmdRename(args))
    case "lsp":
        fs := flag.NewFlagSet("lsp", flag.ExitOnError)
        gopls := fs.String("gopls", "", "the gopls to put behind the server; found when empty, none when \"off\"")
        fs.Parse(args)
        if err := lsp.Serve(os.Stdin, os.Stdout, *gopls); err != nil {
            fmt.Fprintln(os.Stderr, "litgo lsp:", err)
            os.Exit(1)
        }
    case "help", "-h", "--help":
        fmt.Print(usage)
    default:
        fmt.Fprintf(os.Stderr, "litgo: unknown command %q\n\n%s", cmd, usage)
        os.Exit(2)
    }
}
func load(path string) (*lit.Doc, error) {
    abs, err := filepath.Abs(path)
    if err != nil {
        return nil, err
    }
    src, err := os.ReadFile(abs)
    if err != nil {
        return nil, err
    }
    return lit.Parse(abs, src), nil
}

func stdinDoc(path string) *lit.Doc {
    src, _ := io.ReadAll(os.Stdin)
    if path == "" {
        path = "stdin.lit.md"
    }
    return lit.Parse(path, src)
}

func printJSON(v any) int {
    enc := json.NewEncoder(os.Stdout)
    enc.SetEscapeHTML(false)
    if err := enc.Encode(v); err != nil {
        fmt.Fprintln(os.Stderr, "litgo:", err)
        return 1
    }
    return 0
}

func jsonError(err error) int {
    printJSON(map[string]string{"error": err.Error()})
    return 1
}

func report(diags []lit.Diag) (errors int) {
    r := &reporter{seen: map[string]bool{}}
    for _, d := range diags {
        r.diag(d)
        if d.Severity == lit.SevError {
            errors++
        }
    }
    return errors
}
func cmdTangle(args []string) int {
    fs := flag.NewFlagSet("tangle", flag.ExitOnError)
    force := fs.Bool("force", false, "overwrite files litgo did not generate")
    stdout := fs.Bool("stdout", false, "print instead of writing")
    fs.Parse(args)
    if fs.NArg() == 0 {
        fmt.Fprintln(os.Stderr, "litgo tangle: no input files")
        return 2
    }
    status := 0
    for _, path := range fs.Args() {
        doc, err := load(path)
        if err != nil {
            fmt.Fprintln(os.Stderr, "litgo:", err)
            status = 1
            continue
        }
        res := doc.Tangle()
        if report(res.Diags) > 0 {
            status = 1
            continue
        }
        for _, o := range res.Files {
            if *stdout {
                if len(res.Files) > 1 {
                    fmt.Printf("==> %s <==\n", relPath(o.Path))
                }
                os.Stdout.Write(o.Bytes())
                continue
            }
            if err := writeTangled(o, *force); err != nil {
                fmt.Fprintln(os.Stderr, "litgo:", err)
                status = 1
                continue
            }
            fmt.Fprintf(os.Stderr, "litgo: %s → %s (%d lines)\n", relPath(doc.Path), relPath(o.Path), len(o.Lines))
        }
    }
    return status
}

func cmdRun(args []string) int {
    fs := flag.NewFlagSet("run", flag.ExitOnError)
    o := runOptions{}
    fs.BoolVar(&o.json, "json", false, "emit one JSON event per line")
    fs.BoolVar(&o.force, "force", false, "overwrite files litgo did not generate")
    fs.BoolVar(&o.vet, "vet", false, "also run go vet")
    fs.BoolVar(&o.noProve, "no-prove", false, "run even if the //@ annotations are not proved")
    fs.DurationVar(&o.timeout, "timeout", 0, "kill the program after this long")
    fs.Parse(args)
    if fs.NArg() == 0 {
        fmt.Fprintln(os.Stderr, "litgo run: no input file")
        return 2
    }
    o.args = fs.Args()[1:]
    if len(o.args) > 0 && o.args[0] == "--" {
        o.args = o.args[1:]
    }
    return runFile(fs.Arg(0), o)
}

func cmdWeave(args []string) int {
    fs := flag.NewFlagSet("weave", flag.ExitOnError)
    to := fs.String("to", "", "pdf, html, typ (Typst source) or md (woven Markdown); default pdf, or what -o ends in")
    out := fs.String("o", "", "output file")
    fs.Parse(args)
    if fs.NArg() != 1 {
        fmt.Fprintln(os.Stderr, "litgo weave: expected one input file")
        return 2
    }
    doc, err := load(fs.Arg(0))
    if err != nil {
        fmt.Fprintln(os.Stderr, "litgo:", err)
        return 1
    }
    format := *to
    if format == "" {
        format = strings.TrimPrefix(filepath.Ext(*out), ".")
    }
    if format == "" {
        format = "pdf"
    }
    if *out == "" {
        ext := map[string]string{"md": "woven.md"}[format]
        if ext == "" {
            ext = format
        }
        *out = strings.TrimSuffix(strings.TrimSuffix(doc.Path, ".md"), ".lit") + "." + ext
    }
    switch format {
    case "pdf":
        err = weave.PDF(doc, *out)
    case "html":
        err = weave.HTML(doc, *out)
    case "typ":
        var src string
        if src, err = weave.Typst(doc); err == nil {
            err = os.WriteFile(*out, []byte(src), 0o644)
        }
    case "md":
        err = os.WriteFile(*out, []byte(weave.Markdown(doc)), 0o644)
    default:
        err = fmt.Errorf("cannot weave to %q: pdf, html, typ and md are what there is", format)
    }
    if err != nil {
        fmt.Fprintln(os.Stderr, "litgo:", err)
        return 1
    }
    abs, _ := filepath.Abs(*out)
    fmt.Println(abs)
    return 0
}

func cmdCheck(args []string) int {
    status, cache := 0, &check.Cache{}
    for _, path := range args {
        doc, err := load(path)
        if err != nil {
            fmt.Fprintln(os.Stderr, "litgo:", err)
            status = 1
            continue
        }
        res := doc.Tangle()
        diags := append(res.Diags, res.SyntaxCheck()...)
        if len(diags) == len(res.Diags) {
            diags = append(diags, check.Types(res, cache)...)
        }
        if len(diags) == len(res.Diags) {
            proofs, _ := vego.Prove(res)
            diags = append(diags, proofs...)
        }
        if report(diags) > 0 {
            status = 1
        }
    }
    return status
}

prove is the slice of check that cares about the //@ annotations, plus a count at the end. After a proof the news is usually good, and silence can’t tell you that. It checks structure and syntax first, because a proof about a program that doesn’t parse is a proof about nothing. It doesn’t wait for the type checker, though. The verifier reads types straight off the declarations, and a proof is worth having even while an import is still missing.

func cmdProve(args []string) int {
    fs := flag.NewFlagSet("prove", flag.ExitOnError)
    asJSON := fs.Bool("json", false, "print the findings and the count as JSON")
    fs.Parse(args)
    if fs.NArg() == 0 {
        fmt.Fprintln(os.Stderr, "litgo prove: no input files")
        return 2
    }
    status := 0
    for _, path := range fs.Args() {
        doc, err := load(path)
        if err != nil {
            fmt.Fprintln(os.Stderr, "litgo:", err)
            status = 1
            continue
        }
        res := doc.Tangle()
        diags := append(res.Diags, res.SyntaxCheck()...)
        var sum vego.Summary
        if len(diags) == len(res.Diags) && !res.HasErrors() {
            var proofs []lit.Diag
            proofs, sum = vego.Prove(res)
            diags = append(diags, proofs...)
        }
        failed := false
        for _, d := range diags {
            failed = failed || d.Severity == lit.SevError
        }
        if failed {
            status = 1
        }
        if *asJSON {
            if diags == nil {
                diags = []lit.Diag{}
            }
            printJSON(map[string]any{"file": doc.Path, "diagnostics": diags, "functions": sum.Functions,
                "proved": sum.Proved, "obligations": sum.Obligations, "names": sum.Names})
            continue
        }
        report(diags)
        switch {
        case failed && sum.Functions == 0:
        case sum.Functions == 0:
            fmt.Fprintf(os.Stderr, "litgo: %s has no //@ annotations, so there is nothing to prove\n", relPath(doc.Path))
        default:
            fmt.Fprintf(os.Stderr, "litgo: %s: proved %d of %d annotated functions (%d obligations)\n",
                relPath(doc.Path), sum.Proved, sum.Functions, sum.Obligations)
            for _, name := range sum.Names {
                fmt.Fprintf(os.Stderr, "  ✓ %s\n", name)
            }
        }
    }
    return status
}

cat prints a document the way the editor shows it. It’s the quickest way to see what the renderers do.

// cmdCat prints the document the way the editor shows it: math as Unicode,
// diagrams drawn, chunk syntax prettified.
func cmdCat(args []string) int {
    if len(args) != 1 {
        fmt.Fprintln(os.Stderr, "litgo cat: expected one input file")
        return 2
    }
    doc, err := load(args[0])
    if err != nil {
        fmt.Fprintln(os.Stderr, "litgo:", err)
        return 1
    }
    a := doc.Analyze(lit.AnalyzeOptions{})
    blocks := map[int]lit.Item{}
    inline := map[int][]lit.Item{}
    for _, it := range a.Items {
        if it.Kind == "table" || it.Kind == "vego" {
            continue // lining up a table and colouring an annotation are the editor's business
        } else if len(it.Lines) > 0 {
            blocks[it.Line] = it
        } else {
            inline[it.Line] = append(inline[it.Line], it)
        }
    }
    for i := 0; i < len(doc.Lines); i++ {
        if it, ok := blocks[i]; ok {
            for _, l := range it.Lines {
                fmt.Println("    " + l)
            }
            i = it.EndLine
            continue
        }
        line := doc.Lines[i]
        items := inline[i]
        sort.Slice(items, func(a, b int) bool { return items[a].Col < items[b].Col })
        for k := len(items) - 1; k >= 0; k-- { // right to left keeps columns valid
            it := items[k]
            line = line[:it.Col] + it.Text + line[min(it.EndCol, len(line)):]
        }
        fmt.Println(line)
    }
    return 0
}

go/format accepts fragments (a list of statements, not just a whole file), so most blocks can be formatted right where they sit. A block that only parses once its chunks are spliced in gets left alone.

func cmdFmt(args []string) int {
    fs := flag.NewFlagSet("fmt", flag.ExitOnError)
    write := fs.Bool("w", false, "write the result back instead of printing it")
    fs.Parse(args)
    if fs.NArg() != 1 {
        fmt.Fprintln(os.Stderr, "litgo fmt: expected one input file")
        return 2
    }
    doc, err := load(fs.Arg(0))
    if err != nil {
        fmt.Fprintln(os.Stderr, "litgo:", err)
        return 1
    }
    lines := append([]string{}, doc.Lines...)
    for i := len(doc.Blocks) - 1; i >= 0; i-- {
        b := doc.Blocks[i]
        if !b.Tangled() || b.Lang != "go" || b.Last() < b.First() || b.Strip > 0 {
            continue
        }
        // format.Source accepts fragments; blocks that only make sense once
        // their chunks are spliced in fail to parse and are left alone. So are
        // blocks with inline references: gofmt would tidy away the comma that
        // precedes a reference standing for the rest of an argument list.
        if !b.Verbatim && inlineRefs(doc.Lines[b.First():b.Last()+1]) {
            continue
        }
        src := strings.Join(doc.Lines[b.First():b.Last()+1], "\n") + "\n"
        out, err := format.Source([]byte(src))
        if err != nil {
            continue
        }
        formatted := strings.Split(strings.TrimRight(string(out), "\n"), "\n")
        lines = append(append(append([]string{}, lines[:b.First()]...), formatted...), lines[b.Last()+1:]...)
    }
    text := strings.Join(lines, "\n") + "\n"
    if *write {
        if err := os.WriteFile(doc.Path, []byte(text), 0o644); err != nil {
            fmt.Fprintln(os.Stderr, "litgo:", err)
            return 1
        }
        return 0
    }
    fmt.Print(text)
    return 0
}
func inlineRefs(lines []string) bool {
    for i, l := range lines {
        for _, r := range lit.RefsIn(l, i) {
            if r.Inline {
                return true
            }
        }
    }
    return false
}

func cmdAnalyze(args []string) int {
    fs := flag.NewFlagSet("analyze", flag.ExitOnError)
    path := fs.String("path", "", "the document's file name")
    upright := fs.Bool("upright", false, "do not italicise math letters")
    fs.Parse(args)
    return printJSON(stdinDoc(*path).Analyze(lit.AnalyzeOptions{Upright: *upright}))
}

func cmdUntangle(args []string) int {
    fs := flag.NewFlagSet("untangle", flag.ExitOnError)
    path := fs.String("path", "", "the document's file name")
    start := fs.Int("start", -1, "first selected line")
    end := fs.Int("end", -1, "last selected line")
    startCol := fs.Int("start-col", 0, "first selected byte (charwise)")
    endCol := fs.Int("end-col", -1, "byte after the selection (charwise)")
    name := fs.String("name", "", "chunk name; suggested when empty")
    fs.Parse(args)
    res, err := stdinDoc(*path).Untangle(lit.Selection{StartLine: *start, EndLine: *end, StartCol: *startCol, EndCol: *endCol}, *name)
    if err != nil {
        return jsonError(err)
    }
    return printJSON(res)
}

func cmdVersion(args []string) int {
  fmt.Printf("%s\n", VERSION)
  return 0
}

func cmdRename(args []string) int {
    fs := flag.NewFlagSet("rename", flag.ExitOnError)
    path := fs.String("path", "", "the document's file name")
    from := fs.String("from", "", "current chunk name")
    to := fs.String("to", "", "new chunk name")
    fs.Parse(args)
    edits, err := stdinDoc(*path).Rename(*from, *to)
    if err != nil {
        return jsonError(err)
    }
    return printJSON(map[string]any{"edits": edits, "name": lit.NormName(*to)})
}

The feedback loop

run.go

litgo run is the reason everything above exists. It tangles, proves, compiles and runs the program, and reports every position in terms of the document. The stages go in order of cost, and any one of them stops the run if it fails. A missing chunk costs microseconds to find. A broken invariant costs milliseconds. A type error costs a compiler. A wrong answer costs a whole execution, plus somebody paying attention.

sequenceDiagram
    participant E as editor
    participant L as litgo run
    participant G as go build
    participant P as program
    E->>L: pool.lit.md
    L->>L: prove the //@ annotations of pool.go
    L-->>E: pool.lit.md:80:1: Invariant is not maintained
    L->>G: pool.go
    G-->>L: pool.go:31:14: undefined: jobb
    L-->>E: pool.lit.md:67:15: undefined: jobb
    L->>P: start
    P-->>L: output, or a panic with a stack trace
    L-->>E: output, positions translated

Progress goes to a person as text, or to an editor as one JSON event per line: tangle, prove, diagnostic, build, output and exit. A failed proof is a diagnostic, just like a compile error. The only difference is its source, which is vego. An editor can use that to paint the two differently, and the Neovim plugin does.

The reporter has no lock, so only one goroutine gets to use it: the one running runFile. When a program’s output has to be read from two pipes at once, the pipe readers send their lines over a channel to that one goroutine, and it does all the reporting.

// reporter carries the run's progress to a person (plain text) or to an
// editor (one JSON event per line). It belongs to one goroutine.
type reporter struct {
    json bool
    seen map[string]bool
}

func (r *reporter) emit(event string, fields map[string]any) {
    if !r.json {
        return
    }
    fields["event"] = event
    b, _ := json.Marshal(fields)
    fmt.Println(string(b))
}

func (r *reporter) diag(d lit.Diag) {
    key := fmt.Sprintf("%s:%d:%d:%s", d.File, d.Line, d.Col, d.Message)
    if r.seen[key] {
        return
    }
    r.seen[key] = true
    if r.json {
        b, _ := json.Marshal(d)
        var fields map[string]any
        json.Unmarshal(b, &fields)
        r.emit("diagnostic", fields)
        return
    }
    sev := map[int]string{lit.SevError: "error", lit.SevWarning: "warning", lit.SevInfo: "note"}[d.Severity]
    fmt.Fprintf(os.Stderr, "%s:%d:%d: %s: %s\n", relPath(d.File), d.Line+1, d.Col+1, sev, d.Message)
}

func (r *reporter) output(stream, text string) {
    if r.json {
        r.emit("output", map[string]any{"stream": stream, "text": text})
        return
    }
    if stream == "stderr" {
        fmt.Fprintln(os.Stderr, text)
    } else {
        fmt.Println(text)
    }
}

func (r *reporter) status(format string, args ...any) {
    if !r.json {
        fmt.Fprintf(os.Stderr, "litgo: "+format+"\n", args...)
    }
}

func relPath(p string) string {
    if wd, err := os.Getwd(); err == nil {
        if rel, err := filepath.Rel(wd, p); err == nil && !strings.HasPrefix(rel, "..") {
            return rel
        }
    }
    return p
}

Translating positions

Anything that looks like file.go:12:5 and names a file litgo tangled gets rewritten, wherever it shows up: in a compiler error, halfway through a message that mentions a second position, or deep in a stack trace.

// remapper rewrites positions in tangled files back to the literate source.
type remapper struct {
    dir     string
    outputs map[string]*mapped // by absolute output path
}

// mapped is a tangled file together with the document it came from.
type mapped struct {
    *lit.Output
    doc *lit.Doc
}

var posRe = regexp.MustCompile(`((?:[A-Za-z]:)?[\w./\\-]*\.go):(\d+)(?::(\d+))?`)

// lookup finds the tangled file a tool is talking about. Compilers name
// files relative to where they ran; `go test` names them relative to the
// package, so a bare file name is matched if it is unambiguous.
func (m *remapper) lookup(path string) *mapped {
    abs := path
    if !filepath.IsAbs(abs) {
        abs = filepath.Join(m.dir, abs)
    }
    if o := m.outputs[filepath.Clean(abs)]; o != nil || filepath.Base(path) != path {
        return o
    }
    var found *mapped
    for p, o := range m.outputs {
        if filepath.Base(p) == path {
            if found != nil {
                return nil
            }
            found = o
        }
    }
    return found
}

// line rewrites every `file.go:line:col` in a line of tool output and returns
// the first position it translated.
func (m *remapper) line(s string) (string, *lit.Diag) {
    var first *lit.Diag
    out := posRe.ReplaceAllStringFunc(s, func(match string) string {
        g := posRe.FindStringSubmatch(match)
        res := m.lookup(g[1])
        if res == nil {
            return match
        }
        ln, _ := strconv.Atoi(g[2])
        col := 1
        if g[3] != "" {
            col, _ = strconv.Atoi(g[3])
        }
        srcLine, srcCol, _, ok := res.Map(ln-1, col-1)
        if !ok {
            return match
        }
        if first == nil {
            first = &lit.Diag{File: res.doc.Path, Line: srcLine, Col: srcCol}
        }
        if g[3] == "" {
            return fmt.Sprintf("%s:%d", relPath(res.doc.Path), srcLine+1)
        }
        return fmt.Sprintf("%s:%d:%d", relPath(res.doc.Path), srcLine+1, srcCol+1)
    })
    return out, first
}
var compilerLineRe = regexp.MustCompile(`^(?:\./)?([^\s:]+\.go):(\d+)(?::(\d+))?:\s+(.*)$`)

// toolOutput turns compiler or vet output into diagnostics, and reports
// whatever it could not place (linker errors, package-level complaints).
func (m *remapper) toolOutput(out []byte, sev int, source string, r *reporter) {
    var last *lit.Diag
    flush := func() {
        if last != nil {
            r.diag(*last)
            last = nil
        }
    }
    for _, raw := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") {
        if raw == "" || strings.HasPrefix(raw, "# ") {
            continue
        }
        if g := compilerLineRe.FindStringSubmatch(raw); g != nil {
            flush()
            if _, pos := m.line(g[1] + ":" + g[2] + ":" + firstNonEmpty(g[3], "1")); pos != nil {
                msg, _ := m.line(g[4])
                pos.Severity, pos.Message, pos.Source = sev, msg, source
                last = pos
                continue
            }
        }
        if last != nil && (strings.HasPrefix(raw, "\t") || strings.HasPrefix(raw, "  ")) {
            cont, _ := m.line(strings.TrimSpace(raw))
            last.Message += "\n" + cont
            continue
        }
        flush()
        text, _ := m.line(raw)
        r.output("stderr", text)
    }
    flush()
}

func firstNonEmpty(a, b string) string {
    if a != "" {
        return a
    }
    return b
}
// writeTangled writes a file unless that would clobber a hand-written one.
func writeTangled(o *lit.Output, force bool) error {
    if old, err := os.ReadFile(o.Path); err == nil {
        if bytes.Equal(old, o.Bytes()) {
            return nil // leave the timestamp alone so builds stay cached
        }
        if !force && !lit.Generated(old) && len(bytes.TrimSpace(old)) > 0 && lit.CommentsFor(o.File.Lang).Line != "" {
            return fmt.Errorf("%s exists and was not generated by litgo; use --force to overwrite it, or tangle to another <!-- file: ... -->", relPath(o.Path))
        }
    }
    if err := os.MkdirAll(filepath.Dir(o.Path), 0o755); err != nil {
        return err
    }
    return os.WriteFile(o.Path, o.Bytes(), 0o644)
}

The pipeline

type runOptions struct {
    json    bool
    force   bool
    vet     bool
    noProve bool
    timeout time.Duration
    args    []string
}
// runFile is the feedback loop: tangle, compile, run, with every position in
// every message pointing at the .lit.md source.
func runFile(path string, o runOptions) int {
    r := &reporter{json: o.json, seen: map[string]bool{}}
    // <<read the document, and its siblings if it has a run command>>

    m := &remapper{dir: dir, outputs: map[string]*mapped{}}
    // <<tangle and write every document>>
    // <<let the document build itself if it knows how>>

    // <<compile the default file>>

    // <<vet it if asked, and stop here if it is a library>>
    return execute(r, m, exec.Command(bin, o.args...), dir, o.timeout, false)
}
⟨read the document, and its siblings if it has a run command⟩ ≡ used in the program
abs, err := filepath.Abs(path)
if err != nil {
    return fail(r, err)
}
dir := filepath.Dir(abs)
src, err := os.ReadFile(abs)
if err != nil {
    return fail(r, err)
}
doc := lit.Parse(abs, src)

// A custom run command may build a whole package, so tangle the siblings too.
docs := []*lit.Doc{doc}
if doc.Run != "" {
    siblings, _ := filepath.Glob(filepath.Join(dir, "*.lit.md"))
    for _, s := range siblings {
        if s == abs {
            continue
        }
        if b, err := os.ReadFile(s); err == nil {
            docs = append(docs, lit.Parse(s, b))
        }
    }
}

Every file that gets written is registered with the remapper. If the document’s structure is broken, the run stops here. There’s no point compiling half a program.

⟨tangle and write every document⟩ ≡ used in the program
ok, unproved := true, false
for _, d := range docs {
    res := d.Tangle()
    for _, dg := range res.Diags {
        r.diag(dg)
    }
    if res.HasErrors() {
        ok = false
        continue
    }
    for _, out := range res.Files {
        if err := writeTangled(out, o.force); err != nil {
            r.diag(lit.Diag{File: d.Path, Line: max(out.File.Line, 0), Severity: lit.SevError, Message: err.Error(), Source: "litgo"})
            ok = false
            continue
        }
        m.outputs[filepath.Clean(out.Path)] = &mapped{out, d}
    }
    if n := len(res.Files); n == 1 {
        r.emit("tangle", map[string]any{"file": d.Path, "out": res.Files[0].Path})
        r.status("tangled %s → %s", relPath(d.Path), relPath(res.Files[0].Path))
    } else if n > 1 {
        r.emit("tangle", map[string]any{"file": d.Path, "out": fmt.Sprintf("%d files", n)})
        r.status("tangled %s → %d files", relPath(d.Path), n)
    }
    // <<prove what is annotated>>
}
if !ok {
    stage := "tangle"
    if unproved {
        stage = "prove"
        r.status("not proved, so not run (--no-prove runs it anyway)")
    }
    r.emit("build", map[string]any{"ok": false, "ms": 0, "stage": stage})
    return 1
}

A document with //@ annotations gets proved before it gets compiled, on purpose. A proof needs no compiler and takes milliseconds. And a program whose contract doesn’t hold is not a program anyone asked to watch run. The output of a wrong Divide only distracts from the line that says its invariant is not maintained. So a failed proof stops the run, just like a compile error, and is reported the same way: a diagnostic on a line of the Markdown, with vego as its source. --no-prove is for the times when watching it run is how you find out what the invariant should have been.

⟨prove what is annotated⟩ ≡ used in ⟨tangle and write every document⟩
if !o.noProve {
    start := time.Now()
    proofs, sum := vego.Prove(res)
    for _, dg := range proofs {
        r.diag(dg)
        if dg.Severity == lit.SevError {
            ok, unproved = false, true
        }
    }
    if sum.Functions > 0 {
        r.emit("prove", map[string]any{"file": d.Path, "functions": sum.Functions, "proved": sum.Proved,
            "obligations": sum.Obligations, "ms": time.Since(start).Milliseconds()})
        r.status("proved %d of %d annotated functions in %s (%d obligations, %d ms)", sum.Proved, sum.Functions,
            relPath(d.Path), sum.Obligations, time.Since(start).Milliseconds())
    }
}

This document takes the first branch. Its run: directive vets, tests and rebuilds litgo.

⟨let the document build itself if it knows how⟩ ≡ used in the program
// A document that writes several files is a package or a module, and
// only it knows how it is built
command := doc.Run
if command == "" && len(doc.Files) > 1 {
    command = "go build ./..."
}
if command != "" {
    r.status("$ %s", command)
    return execute(r, m, exec.Command("sh", "-c", command), dir, o.timeout, true)
}
def := doc.Files[0]
main := m.outputs[filepath.Clean(doc.OutPath(def))]
if def.Lang != "go" || main == nil {
    r.status("no run step for tangler %q; add <!-- run: command -->", def.Lang)
    r.emit("build", map[string]any{"ok": true, "ms": 0})
    return 0
}

Building and running are separate steps. That way litgo can tell a compile error from a program that exits non-zero, and it can time each one.

⟨compile the default file⟩ ≡ used in the program
tmp, err := os.MkdirTemp("", "litgo-run")
if err != nil {
    return fail(r, err)
}
defer os.RemoveAll(tmp)
bin := filepath.Join(tmp, "prog")

start := time.Now()
build := exec.Command("go", /*<<arguments to go build>>*/)
build.Dir = dir
out, err := build.CombinedOutput()
ms := time.Since(start).Milliseconds()
m.toolOutput(out, lit.SevError, "go build", r)
if err != nil {
    r.emit("build", map[string]any{"ok": false, "ms": ms, "stage": "compile"})
    r.status("build failed (%d ms)", ms)
    return 1
}
r.emit("build", map[string]any{"ok": true, "ms": ms})
r.status("built in %d ms", ms)

The compiler gives up after ten errors unless you tell it not to. In a tight loop you want to see all of them.

⟨arguments to go build⟩ ≡ used in ⟨compile the default file⟩
"build", "-gcflags=-e", "-o", bin, filepath.Base(main.Path)
⟨vet it if asked, and stop here if it is a library⟩ ≡ used in the program
if o.vet {
    vet := exec.Command("go", "vet", filepath.Base(main.Path))
    vet.Dir = dir
    out, _ := vet.CombinedOutput()
    m.toolOutput(out, lit.SevWarning, "go vet", r)
}
if def.Package != "main" && def.Package != "" {
    r.status("package %s compiles; nothing to run", def.Package)
    r.emit("exit", map[string]any{"code": 0, "ms": 0, "ran": false})
    return 0
}

Running

Output streams line by line, and positions get translated on the way through. A panic becomes a diagnostic on the innermost stack frame that lives in the document. So does a failing test, when the document’s own command runs go test.

The program’s stdout and stderr both have to be read while it runs, so each pipe gets a goroutine. Those goroutines report nothing themselves. They send every line down one channel, and the loop in execute receives them and does the reporting. The same loop watches the timeout. So the reporter, the panic bookkeeping and the timedOut flag all belong to one goroutine, and none of them needs a lock.

flowchart LR
    out[read stdout] --> loop[the loop in execute]
    err[read stderr] --> loop
    timer[timeout] --> loop
    loop --> reporter
func fail(r *reporter, err error) int {
    r.diag(lit.Diag{Severity: lit.SevError, Message: err.Error(), Source: "litgo"})
    return 1
}

var (
    frameRe = regexp.MustCompile(`^\s+(\S+\.go):(\d+)`)
    // reportRe is looser than compilerLineRe: `go test` indents its failures.
    reportRe = regexp.MustCompile(`^\s*(?:\./)?[^\s:]+\.go:\d+(?::\d+)?:\s+(.*)$`)
)

// outputLine is one line of a program's output, or the end of a stream.
type outputLine struct {
    stream, text string
    eof          bool
}

// pump sends the lines of one of a program's streams, and then its end.
func pump(stream string, rd io.Reader, lines chan<- outputLine) {
    sc := bufio.NewScanner(rd)
    sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
    for sc.Scan() {
        lines <- outputLine{stream: stream, text: sc.Text()}
    }
    lines <- outputLine{stream: stream, eof: true}
}

// execute streams a program's output, translating positions as it goes. A
// panic becomes a diagnostic on the innermost frame that lives in the
// literate source.
func execute(r *reporter, m *remapper, cmd *exec.Cmd, dir string, timeout time.Duration, parseTools bool) int {
    cmd.Dir = dir
    stdout, _ := cmd.StdoutPipe()
    stderr, _ := cmd.StderrPipe()
    ownGroup(cmd)
    start := time.Now()
    if err := cmd.Start(); err != nil {
        return fail(r, err)
    }
    defer killOnSignal(cmd)()
    var timeUp <-chan time.Time // stays nil, which never delivers, without a timeout
    if timeout > 0 {
        timer := time.NewTimer(timeout)
        defer timer.Stop()
        timeUp = timer.C
    }

    lines := make(chan outputLine)
    go pump("stdout", stdout, lines)
    go pump("stderr", stderr, lines)
    timedOut := false
    var panicMsg string
    panicPlaced := false
    for open := 2; open > 0; {
        select {
        case <-timeUp:
            timedOut = true
            killGroup(cmd)
        case l := <-lines:
            if l.eof {
                open--
                continue
            }
            raw := l.text
            text, pos := m.line(raw)
            if l.stream == "stderr" {
                switch {
                case strings.HasPrefix(raw, "panic: ") || strings.HasPrefix(raw, "fatal error: "):
                    panicMsg, panicPlaced = raw, false
                case panicMsg != "" && !panicPlaced && pos != nil && frameRe.MatchString(raw):
                    pos.Severity, pos.Message, pos.Source = lit.SevError, panicMsg, "runtime"
                    r.diag(*pos)
                    panicPlaced = true
                case parseTools && pos != nil && reportRe.MatchString(raw):
                    msg, _ := m.line(reportRe.FindStringSubmatch(raw)[1])
                    pos.Severity, pos.Message, pos.Source = lit.SevError, msg, "run"
                    r.diag(*pos)
                }
            } else if parseTools && pos != nil && reportRe.MatchString(raw) {
                msg, _ := m.line(reportRe.FindStringSubmatch(raw)[1])
                pos.Severity, pos.Message, pos.Source = lit.SevError, msg, "run"
                r.diag(*pos)
            }
            r.output(l.stream, text)
        }
    }
    err := cmd.Wait()
    code := 0
    if err != nil {
        code = 1
        if ee, ok := err.(*exec.ExitError); ok && ee.ExitCode() >= 0 {
            code = ee.ExitCode()
        }
    }
    ms := time.Since(start).Milliseconds()
    if timedOut {
        r.output("stderr", fmt.Sprintf("litgo: killed after %s", timeout))
    }
    r.emit("exit", map[string]any{"code": code, "ms": ms, "ran": true, "timed_out": timedOut})
    r.status("exit %d (%d ms)", code, ms)
    return code
}

The program runs in its own process group. Stop litgo and you stop the program and everything it started. The timeout does the same.

proc_unix.go
//go:build unix

package main

import (
    "os"
    "os/exec"
    "os/signal"
    "syscall"
)

// ownGroup puts the child in its own process group so that it, and anything
// it spawns, can be killed as one.
func ownGroup(cmd *exec.Cmd) {
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
}

func killGroup(cmd *exec.Cmd) {
    if cmd.Process != nil {
        syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
    }
}

// killOnSignal makes stopping litgo (the editor's :LitStop) stop the program
// it is running. The returned function ends the watch.
func killOnSignal(cmd *exec.Cmd) func() {
    sigs := make(chan os.Signal, 1)
    done := make(chan struct{})
    signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
    go func() {
        select {
        case <-sigs:
            killGroup(cmd)
        case <-done:
        }
    }()
    return func() {
        signal.Stop(sigs)
        close(done)
    }
}
proc_other.go
//go:build !unix

package main

import "os/exec"

func ownGroup(cmd *exec.Cmd) {}

func killGroup(cmd *exec.Cmd) {
    if cmd.Process != nil {
        cmd.Process.Kill()
    }
}

func killOnSignal(cmd *exec.Cmd) func() { return func() {} }

Errors as you type

litgo run finds every error, but only when you ask, and only after it has written files and started a compiler. Most errors can be caught much sooner. The Go type checker lives in the standard library. Hand it the tangled files in memory and it reports what the compiler would (an undefined name, a wrong argument, an unused variable, an unused import) in a few milliseconds. The source map puts each report on the right line of the Markdown. A language server runs that check after every change, so errors show up under the code that caused them before you’ve even thought about running anything. The verifier works the same way, on the same files in memory, so a document with //@ annotations gets proved after every change too.

sequenceDiagram
    participant E as editor
    participant S as litgo lsp
    participant T as go/types
    E->>S: didChange: the whole buffer
    S-->>E: chunk and syntax errors, at once
    S->>T: the tangled files, never written
    T-->>S: pool.go:31:14: undefined: jobb
    S-->>E: pool.lit.md:67:15: undefined: jobb
    S->>S: prove the //@ annotations, once the types are right
    S-->>E: pool.lit.md:80:1: Invariant is not maintained

Type-checking what is not on disk

internal/check/check.go

A check works on packages, and a package is the document’s outputs in one directory. When the document writes a module or has its own run: command, the other Go files already sitting in that directory come along too. A document that is a single program gets checked alone, the same way litgo run compiles it alone. Who cares? Anyone with a directory of examples, which is full of main functions that can’t share a package.

// pkg is one package about to be checked.
type pkg struct {
    dir, name, path string
    files           []*ast.File // what an importer of the package sees
    tests           []*ast.File // _test.go files of the same package
    types           *types.Package
    checking        bool
}

type checker struct {
    res       *lit.Result
    fset      *token.FileSet
    outputs   map[string]*lit.Output // by path
    pkgs      []*pkg
    byPath    map[string]*pkg   // packages this document writes, by import path
    exports   map[string]string // import path to export data file, for all others
    gc        types.Importer
    goVersion string
    seen      map[string]bool
    diags     []lit.Diag
}

// Cache carries what is worth keeping between checks: where the compiled
// standard library lives. It has no lock, so whoever owns a Cache runs one
// check at a time with it.
type Cache struct {
    std map[string]string
}

// Types type-checks the Go files a document tangles to, without writing
// them, and reports errors at their positions in the document. It reports
// nothing if the files do not parse (the syntax check covers that) or if
// there is no go command to ask about imports.
func Types(res *lit.Result, cache *Cache) []lit.Diag {
    if res.HasErrors() {
        return nil
    }
    c := &checker{res: res, fset: token.NewFileSet(), outputs: map[string]*lit.Output{},
        byPath: map[string]*pkg{}, exports: map[string]string{}, seen: map[string]bool{}}
    if !c.parse() {
        return nil
    }
    c.locate()
    if !c.resolve(cache) {
        return nil
    }
    c.gc = importer.ForCompiler(c.fset, "gc", c.lookup)
    for _, p := range c.pkgs {
        c.check(p, true)
    }
    return c.diags
}

Build constraints matter here. litgo itself has two files that define the same functions for different operating systems. go/build decides which files belong, and it reads the in-memory content through its OpenFile hook.

// parse sorts the files into packages. It fails if an output does not parse.
func (c *checker) parse() bool {
    doc := c.res.Doc
    for _, o := range c.res.Files {
        if o.File.Lang == "go" {
            c.outputs[o.Path] = o
        }
    }
    ctx := build.Default
    ctx.OpenFile = func(path string) (io.ReadCloser, error) {
        if o := c.outputs[path]; o != nil {
            return io.NopCloser(bytes.NewReader(o.Bytes())), nil
        }
        return os.Open(path)
    }
    groups := map[string]*pkg{}
    add := func(path string, src any) error {
        dir, base := filepath.Dir(path), filepath.Base(path)
        if ok, _ := ctx.MatchFile(dir, base); !ok {
            return nil
        }
        f, err := parser.ParseFile(c.fset, path, src, parser.SkipObjectResolution)
        if err != nil {
            return err
        }
        test := strings.HasSuffix(base, "_test.go")
        name := f.Name.Name
        p := groups[dir+"\x00"+name]
        if p == nil {
            p = &pkg{dir: dir, name: name}
            groups[dir+"\x00"+name] = p
            c.pkgs = append(c.pkgs, p)
        }
        if test && !strings.HasSuffix(name, "_test") {
            p.tests = append(p.tests, f)
        } else {
            p.files = append(p.files, f)
        }
        return nil
    }

    dirs := map[string]bool{}
    for path, o := range c.outputs {
        if add(path, o.Bytes()) != nil {
            return false
        }
        dirs[filepath.Dir(path)] = true
    }
    if doc.Run != "" || len(doc.Files) > 1 {
        // <<add the files that are already in those directories>>
    }
    sort.Slice(c.pkgs, func(i, j int) bool {
        return c.pkgs[i].dir+c.pkgs[i].name < c.pkgs[j].dir+c.pkgs[j].name
    })
    return true
}

If a hand-written file doesn’t parse, that’s its author’s problem, not this document’s. The file gets left out.

⟨add the files that are already in those directories⟩ ≡ used in the program
for dir := range dirs {
    entries, _ := os.ReadDir(dir)
    for _, e := range entries {
        path := filepath.Join(dir, e.Name())
        if e.IsDir() || !strings.HasSuffix(path, ".go") || c.outputs[path] != nil {
            continue
        }
        add(path, nil)
    }
}

The packages a document writes can import each other. litgo’s main, for example, imports its internal/lit. Those imports have to resolve to the version in the buffer, not to whatever was tangled last, so each package needs its import path (the module path plus the directory). The go.mod can be one of the outputs too.

// locate works out the import path of every package.
func (c *checker) locate() {
    modDir, mod := "", ""
    for _, o := range c.res.Files {
        if filepath.Base(o.Path) == "go.mod" {
            modDir, mod = filepath.Dir(o.Path), string(o.Bytes())
        }
    }
    for dir := filepath.Dir(c.res.Doc.Path); mod == ""; dir = filepath.Dir(dir) {
        if b, err := os.ReadFile(filepath.Join(dir, "go.mod")); err == nil {
            modDir, mod = dir, string(b)
        }
        if dir == filepath.Dir(dir) {
            break
        }
    }
    module := ""
    for _, line := range strings.Split(mod, "\n") {
        if f := strings.Fields(line); len(f) == 2 && f[0] == "module" {
            module = strings.Trim(f[1], `"`)
        } else if len(f) == 2 && f[0] == "go" {
            c.goVersion = "go" + f[1]
        }
    }
    for _, p := range c.pkgs {
        p.path = p.name
        if rel, err := filepath.Rel(modDir, p.dir); module != "" && err == nil && !strings.HasPrefix(rel, "..") {
            p.path = strings.TrimSuffix(module+"/"+filepath.ToSlash(rel), "/.")
            if strings.HasSuffix(p.name, "_test") {
                p.path += "_test"
            }
        }
        if !strings.HasSuffix(p.name, "_test") && p.name != "main" {
            c.byPath[p.path] = p
        }
    }
}

Everything else gets imported the way go vet does it, from the export data the compiler left in the build cache (a compact summary of a compiled package’s types). go list -export finds that data, compiling first if it has to. It’s exact, and it’s fast. Reading the export data of fmt takes a fraction of the time it would take to type-check its source. One go list call covers all the imports of a check. The standard library is remembered for the life of the process, because it doesn’t change while an editor is running.

// resolve finds the export data of every package that is imported but not
// written by this document. It fails if the go command cannot be run.
func (c *checker) resolve(cache *Cache) bool {
    if cache.std == nil {
        cache.std = map[string]string{}
    }
    var missing []string
    for _, p := range c.pkgs {
        for _, f := range append(append([]*ast.File{}, p.files...), p.tests...) {
            for _, imp := range f.Imports {
                path := strings.Trim(imp.Path.Value, "\"`")
                if c.byPath[path] != nil || path == "C" || path == "unsafe" || c.seen[path] {
                    continue
                }
                c.seen[path] = true
                if file, ok := cache.std[path]; ok && exists(file) {
                    c.exports[path] = file
                } else {
                    missing = append(missing, path)
                }
            }
        }
    }
    c.seen = map[string]bool{}
    if len(missing) == 0 {
        return true
    }
    args := append([]string{"list", "-e", "-export", "-deps", "-f", "{{if .Export}}{{.ImportPath}}\t{{.Export}}{{end}}", "--"}, missing...)
    cmd := exec.Command("go", args...)
    cmd.Dir = filepath.Dir(c.res.Doc.Path)
    out, err := cmd.Output()
    if _, broken := err.(*exec.Error); broken {
        return false
    }
    for _, line := range strings.Split(string(out), "\n") {
        if path, file, ok := strings.Cut(line, "\t"); ok {
            c.exports[path] = file
            if !strings.Contains(strings.Split(path, "/")[0], ".") {
                cache.std[path] = file
            }
        }
    }
    return true
}

func exists(file string) bool {
    _, err := os.Stat(file)
    return err == nil
}

func (c *checker) lookup(path string) (io.ReadCloser, error) {
    if file := c.exports[path]; file != "" {
        return os.Open(file)
    }
    return nil, os.ErrNotExist
}

The checker is its own importer. A package from this document gets checked on demand, once, without its tests. Anything else comes from the export data.

// Import implements types.Importer.
func (c *checker) Import(path string) (*types.Package, error) {
    if p := c.byPath[path]; p != nil {
        return c.check(p, false)
    }
    return c.gc.Import(path)
}

// check type-checks a package: as a root, with its tests and with its
// errors reported, or as the import of another package, silently.
func (c *checker) check(p *pkg, root bool) (*types.Package, error) {
    files := p.files
    if root {
        files = append(append([]*ast.File{}, p.files...), p.tests...)
    } else if p.types != nil {
        return p.types, nil
    } else if p.checking {
        return nil, os.ErrInvalid // an import cycle; the root check of it says so
    }
    conf := types.Config{Importer: c, GoVersion: c.goVersion, FakeImportC: true, Error: func(err error) {
        if root {
            c.report(err)
        }
    }}
    p.checking = true
    tp, _ := conf.Check(p.path, c.fset, files, nil)
    p.checking = false
    if !root {
        p.types = tp
    }
    return tp, nil
}

A chunk used twice with an error in it gets reported once, not twice.

func (c *checker) report(err error) {
    te, ok := err.(types.Error)
    if !ok || len(c.diags) >= 100 {
        return
    }
    pos := c.fset.Position(te.Pos)
    o := c.outputs[pos.Filename]
    if o == nil {
        return
    }
    line, col, _, ok := o.Map(pos.Line-1, pos.Column-1)
    key := fmt.Sprintf("%d:%d:%s", line, col, te.Msg)
    if !ok || c.seen[key] {
        return
    }
    c.seen[key] = true
    c.diags = append(c.diags, lit.Diag{File: c.res.Doc.Path, Line: line, Col: col, Severity: lit.SevError, Message: te.Msg, Source: "types"})
}
internal/check/check_test.go

The test document writes a small module: a library, a program that uses it, and a test. None of it exists on disk, and each error hides in a different kind of place.

const module = "<!-- file: go.mod -->\n```gomod\nmodule example.com/m\n\ngo 1.22\n```\n\n" +
    "<!-- file: lib/lib.go -->\n<!-- package: lib -->\n<!-- imports: os, strings -->\n" +
    "```go\nfunc Shout(s string) string {\n\t// <<upper>>\n}\n```\n\n" +
    "<!-- chunk: upper -->\n```go\nreturn strings.ToUpper(s) + suffix\n```\n\n" +
    "<!-- file: lib/lib_test.go -->\n<!-- package: lib -->\n<!-- imports: testing -->\n" +
    "```go\nfunc TestShout(t *testing.T) {\n\tif Shout(1) != \"A\" {\n\t\tt.Fail()\n\t}\n}\n```\n\n" +
    "<!-- file: main.go -->\n<!-- package: main -->\n<!-- imports: fmt, example.com/m/lib -->\n" +
    "```go\nfunc main() {\n\tn := 1\n\tfmt.Println(lib.Shout(\"a\"), lib.Whisper)\n}\n```\n"

func TestTypes(t *testing.T) {
    path := filepath.Join(t.TempDir(), "m.lit.md")
    os.WriteFile(path, []byte(module), 0o644)
    doc := lit.Parse(path, []byte(module))
    diags := Types(doc.Tangle(), &Cache{})

    var got []string
    for _, d := range diags {
        got = append(got, strings.TrimSpace(doc.Lines[d.Line][d.Col:])+" ← "+d.Message)
    }
    for _, want := range []string{
        `os, strings --> ← "os" imported and not used`, // on its entry in the directive
        "suffix ← undefined: suffix",                   // inside a chunk
        "1) != \"A\" { ← cannot use 1",                 // in a test file of the package
        "n := 1 ← declared and not used: n",
        "Whisper) ← undefined: lib.Whisper", // through an import of a package that is not on disk
    } {
        text, msg, _ := strings.Cut(want, " ← ")
        found := false
        for _, g := range got {
            found = found || strings.HasPrefix(g, text) && strings.Contains(g, msg)
        }
        if !found {
            t.Errorf("missing %q in:\n%s", want, strings.Join(got, "\n"))
        }
    }
    if len(diags) != 5 {
        t.Errorf("%d diagnostics, want 5:\n%s", len(diags), strings.Join(got, "\n"))
    }
}

func TestSingleProgramStandsAlone(t *testing.T) {
    dir := t.TempDir()
    os.WriteFile(filepath.Join(dir, "other.go"), []byte("package main\n\nfunc main() {}\n"), 0o644)
    src := "<!-- package: main -->\n```go\nfunc main() {}\n```\n"
    doc := lit.Parse(filepath.Join(dir, "one.lit.md"), []byte(src))
    if diags := Types(doc.Tangle(), &Cache{}); len(diags) != 0 {
        t.Errorf("a second main in the directory is not this program's business: %+v", diags)
    }
}

The language server

internal/lsp/lsp.go

The Language Server Protocol is JSON-RPC over standard input and output, with a Content-Length header in front of each message. The part litgo needs to publish diagnostics is small enough to write out right here.

type message struct {
    JSONRPC string           `json:"jsonrpc"`
    ID      *json.RawMessage `json:"id,omitempty"`
    Method  string           `json:"method,omitempty"`
    Params  json.RawMessage  `json:"params,omitempty"`
    Result  any              `json:"result,omitempty"`
    Error   *rpcError        `json:"error,omitempty"`
}

type rpcError struct {
    Code    int    `json:"code"`
    Message string `json:"message"`
}

func read(r *bufio.Reader) (*message, error) {
    length := 0
    for {
        line, err := r.ReadString('\n')
        if err != nil {
            return nil, err
        }
        line = strings.TrimSpace(line)
        if line == "" {
            break
        }
        if name, value, ok := strings.Cut(line, ":"); ok && strings.EqualFold(name, "Content-Length") {
            length, _ = strconv.Atoi(strings.TrimSpace(value))
        }
    }
    body := make([]byte, length)
    if _, err := io.ReadFull(r, body); err != nil {
        return nil, err
    }
    m := &message{}
    return m, json.Unmarshal(body, m)
}

// frame spells a message the way it goes over the wire.
func frame(m *message) []byte {
    m.JSONRPC = "2.0"
    body, _ := json.Marshal(m)
    return []byte(fmt.Sprintf("Content-Length: %d\r\n\r\n%s", len(body), body))
}

The server has no locks. One goroutine, the loop in Serve, owns all of its state: the open documents, the overlays, and the bookkeeping for gopls. Every other goroutine does one slow or blocking job and talks to the loop over a channel. One reads messages from the editor. One writes messages to the editor. Each changed document has a timer that tells the loop when the typing has paused. At most one goroutine runs the type checker. Only the loop touches the state, so nothing needs a mutex.

flowchart LR
    reader[read the editor] --> loop
    timers --> loop
    checker[type checker] --> loop
    loop --> writer[write to the editor]

A pipe is written by the one goroutine that owns it. Whoever sends a message frames it first, so the writer never holds a value someone else might still change. The channel has a buffer, so the loop keeps going even when the reader on the far end of the pipe is slow. Closing the channel means there’s nothing more to write, and done closes once the last message is out.

// writer starts the goroutine that writes to a pipe.
func writer(w io.Writer) (messages chan []byte, done chan struct{}) {
    messages, done = make(chan []byte, 64), make(chan struct{})
    go func() {
        defer close(done)
        for b := range messages {
            w.Write(b)
        }
    }()
    return messages, done
}

func (s *server) write(m *message) { s.out <- frame(m) }

The server keeps the text of every open document, and the whole text arrives with each change. A change starts a short timer, not an analysis, so a burst of typing costs one analysis instead of one per keystroke.

type document struct {
    text    string
    version int
    timer   *time.Timer
    res     *lit.Result // what the text tangles to,
    tangled int         // and the version of the text that was tangled
}

// server is the state of the loop in Serve. Only that goroutine touches it.
type server struct {
    out   chan<- []byte // to the goroutine that writes to the editor
    docs  map[string]*document
    utf8  bool // the client counts columns in bytes, as litgo does
    delay time.Duration

    due      chan due      // a document's timer went off
    checked  chan *job     // the type checker is done with a job
    quit     chan struct{} // closed when the server stops
    waiting  map[string]*job
    checking bool // the type checker is busy, and has the cache
    cache    check.Cache

    // The rest is for the chapter after this one.
    goplsBin string
    gopls    *gopls
    overlays map[string]*overlay // by path of the tangled file
}

// due says that a document has not changed for a while since this version.
type due struct {
    uri     string
    version int
}

// Serve speaks the protocol until the client says exit or hangs up. It puts
// the gopls named (or found, if none is named; "off" means none) behind it.
func Serve(in io.Reader, out io.Writer, gopls string) error {
    toEditor, written := writer(out)
    s := &server{out: toEditor, docs: map[string]*document{}, delay: 120 * time.Millisecond,
        due: make(chan due), checked: make(chan *job), quit: make(chan struct{}), waiting: map[string]*job{},
        goplsBin: gopls, overlays: map[string]*overlay{}}
    defer func() {
        close(s.quit)
        s.stop()
        close(toEditor)
        <-written
    }()
    editor, failed := make(chan *message), make(chan error, 1)
    go func() {
        r := bufio.NewReader(in)
        for {
            m, err := read(r)
            if err != nil {
                failed <- err
                return
            }
            select {
            case editor <- m:
            case <-s.quit:
                return
            }
        }
    }()
    for {
        select {
        case m := <-editor:
            if m.Method == "exit" {
                return nil
            }
            s.handle(m)
        case err := <-failed:
            if err == io.EOF {
                return nil
            }
            return err
        case d := <-s.due:
            if _, now := s.current(d.uri); now == d.version {
                s.analyze(d.uri)
            }
        case j := <-s.checked:
            s.finish(j)
        case m, ok := <-s.heard():
            if ok {
                s.gopls.hear(m, s)
            } else {
                s.gopls.hangUp()
            }
        }
    }
}

Every goroutine that sends to the loop also watches quit, so none of them is left waiting forever on a loop that has gone away.

func (s *server) handle(m *message) {
    switch m.Method {
    case "initialize":
        // <<agree on how columns are counted, and say what the server does>>
    case "shutdown":
        s.write(&message{ID: m.ID, Result: json.RawMessage("null")})
    case "textDocument/didOpen":
        var p struct {
            TextDocument struct {
                URI     string `json:"uri"`
                Text    string `json:"text"`
                Version int    `json:"version"`
            } `json:"textDocument"`
        }
        json.Unmarshal(m.Params, &p)
        s.change(p.TextDocument.URI, p.TextDocument.Text, p.TextDocument.Version)
    case "textDocument/didChange":
        var p struct {
            TextDocument struct {
                URI     string `json:"uri"`
                Version int    `json:"version"`
            } `json:"textDocument"`
            ContentChanges []struct {
                Text string `json:"text"`
            } `json:"contentChanges"`
        }
        json.Unmarshal(m.Params, &p)
        if n := len(p.ContentChanges); n > 0 {
            s.change(p.TextDocument.URI, p.ContentChanges[n-1].Text, p.TextDocument.Version)
        }
    case "textDocument/didClose":
        var p struct {
            TextDocument struct {
                URI string `json:"uri"`
            } `json:"textDocument"`
        }
        json.Unmarshal(m.Params, &p)
        if d := s.docs[p.TextDocument.URI]; d != nil && d.timer != nil {
            d.timer.Stop()
        }
        delete(s.docs, p.TextDocument.URI)
        delete(s.waiting, p.TextDocument.URI)
        s.forget(p.TextDocument.URI)
    default:
        if !s.proxy(m) && m.ID != nil { // a request for something this server does not do
            s.write(&message{ID: m.ID, Error: &rpcError{-32601, "litgo lsp does not implement " + m.Method}})
        }
    }
}

The protocol counts columns in UTF-16 code units unless both sides agree on something else. (Yes, UTF-16. It’s a JavaScript thing.) litgo counts bytes, and so does Neovim, which offers utf-8. For a client that doesn’t, litgo converts columns on the way out. Documents are synchronised whole (change: 1), because they’re small and litgo parses the whole thing anyway.

⟨agree on how columns are counted, and say what the server does⟩ ≡ used in the program
var p struct {
    Capabilities struct {
        General struct {
            PositionEncodings []string `json:"positionEncodings"`
        } `json:"general"`
    } `json:"capabilities"`
}
json.Unmarshal(m.Params, &p)
s.start(m.Params)
encoding := "utf-16"
for _, e := range p.Capabilities.General.PositionEncodings {
    if e == "utf-8" && s.gopls == nil { // gopls counts in UTF-16, and that settles it
        encoding, s.utf8 = e, true
    }
}
capabilities := s.capabilities()
capabilities["positionEncoding"] = encoding
capabilities["textDocumentSync"] = map[string]any{"openClose": true, "change": 1}
s.write(&message{ID: m.ID, Result: map[string]any{
    "serverInfo":   map[string]any{"name": "litgo"},
    "capabilities": capabilities,
}})
func (s *server) change(uri, text string, version int) {
    if !strings.HasSuffix(uri, ".lit.md") {
        return
    }
    d := s.docs[uri]
    if d == nil {
        d = &document{}
        s.docs[uri] = d
    }
    if d.timer != nil {
        d.timer.Stop()
    }
    d.text, d.version = text, version
    d.timer = time.AfterFunc(s.delay, func() {
        select {
        case s.due <- due{uri, version}:
        case <-s.quit:
        }
    })
}

// current returns the text and version of a document, or version -1.
func (s *server) current(uri string) (string, int) {
    if d := s.docs[uri]; d != nil {
        return d.text, d.version
    }
    return "", -1
}

Diagnostics go out in two rounds. Problems with the document’s structure and with Go’s syntax take microseconds to find, so the loop finds them itself and publishes them right away. If there are none, the document needs a type check. That can take a while (the first one in a session has to wait for go list), so it runs in a goroutine of its own and the loop stays free to answer the editor.

Only one type check runs at a time. The cache isn’t locked, and a machine has better things to do than check five versions of one document at once. While the checker is busy, the loop keeps the newest job for each document in waiting, and a newer job replaces an older one. When the checker hands its job back, the loop publishes the findings (unless the document changed in the meantime) and starts the next job.

A document with //@ annotations gets a third round, in the same goroutine, once the types are in order: the proofs. A proof about a program with a type error would be a second opinion on code that’s about to change, so it waits. It costs milliseconds, and that changes everything. Proving becomes something that happens as you type, not something you have to remember to do. Strengthen an invariant, and the violet line under the Ensures vanishes before you’ve reached for a command.

// job is a type check: what to check, and then what the check found. The
// loop and the checker pass it back and forth and never hold it together.
type job struct {
    uri     string
    version int
    res     *lit.Result
    diags   []lit.Diag // published already
    typed   []lit.Diag // what the type checker adds
}

func (s *server) analyze(uri string) {
    res, version := s.sync(uri)
    if res == nil {
        return
    }
    diags := append(append([]lit.Diag{}, res.Diags...), res.SyntaxCheck()...)
    s.publish(uri, version, res.Doc, diags)
    if res.HasErrors() || len(diags) > len(res.Diags) {
        return
    }
    s.waiting[uri] = &job{uri: uri, version: version, res: res, diags: diags}
    s.check()
}

// check gives the type checker its next job, if it is free and there is one
// that is still worth doing. The cache goes to the checker with the job and
// comes back with it.
func (s *server) check() {
    if s.checking {
        return
    }
    for uri, j := range s.waiting {
        delete(s.waiting, uri)
        if _, now := s.current(uri); now != j.version {
            continue
        }
        s.checking = true
        go func() {
            j.typed = check.Types(j.res, &s.cache)
            if len(j.typed) == 0 {
                j.typed, _ = vego.Prove(j.res)
            }
            select {
            case s.checked <- j:
            case <-s.quit:
            }
        }()
        return
    }
}

// finish publishes what a type check found, if it is still true.
func (s *server) finish(j *job) {
    s.checking = false
    if _, now := s.current(j.uri); now == j.version && len(j.typed) > 0 {
        s.publish(j.uri, j.version, j.res.Doc, append(j.diags, j.typed...))
    }
    s.check()
}

func path(uri string) string {
    if u, err := url.Parse(uri); err == nil && u.Scheme == "file" {
        return u.Path
    }
    return uri
}

A diagnostic from litgo is a point, but an editor underlines a range. So litgo stretches the range over the identifier at that point, if there is one.

func (s *server) publish(uri string, version int, doc *lit.Doc, diags []lit.Diag) {
    type position struct {
        Line      int `json:"line"`
        Character int `json:"character"`
    }
    type diagnostic struct {
        Range struct {
            Start position `json:"start"`
            End   position `json:"end"`
        } `json:"range"`
        Severity int    `json:"severity"`
        Source   string `json:"source"`
        Message  string `json:"message"`
    }
    out := []diagnostic{}
    for _, d := range diags {
        line := ""
        if d.Line >= 0 && d.Line < len(doc.Lines) {
            line = doc.Lines[d.Line]
        }
        start := min(max(d.Col, 0), len(line))
        end := d.EndCol
        if end <= start {
            end = start
            for end < len(line) && (line[end] == '_' || line[end] >= '0' && line[end] <= '9' || line[end] >= 'A' && line[end] <= 'Z' || line[end] >= 'a' && line[end] <= 'z' || line[end] >= 0x80) {
                end++
            }
            if end == start {
                end = min(start+1, len(line))
            }
        }
        end = min(end, len(line))
        var ld diagnostic
        ld.Range.Start = position{d.Line, s.column(line, start)}
        ld.Range.End = position{d.Line, s.column(line, end)}
        ld.Severity, ld.Source, ld.Message = d.Severity, "litgo", d.Message
        if d.Source != "" && d.Source != "litgo" {
            ld.Source = "litgo " + d.Source
        }
        out = append(out, ld)
    }
    params, _ := json.Marshal(map[string]any{"uri": uri, "version": version, "diagnostics": out})
    s.write(&message{Method: "textDocument/publishDiagnostics", Params: params})
}

// column converts a byte offset into the client's unit.
func (s *server) column(line string, offset int) int {
    return units(s.utf8, line, offset)
}

// units converts a byte offset in a line to UTF-16 code units, unless bytes
// are what is wanted.
func units(utf8 bool, line string, offset int) int {
    offset = min(max(offset, 0), len(line))
    if utf8 {
        return offset
    }
    n := 0
    for _, r := range line[:offset] {
        n++
        if r >= 0x10000 {
            n++
        }
    }
    return n
}
internal/lsp/lsp_test.go

The test plays the client. It opens a document with a type error in a chunk, fixes the error, and watches the diagnostics appear and then disappear.

const program = "<!-- package: main -->\n<!-- imports: fmt -->\n\n```go\nfunc main() {\n\t// <<greet>>\n}\n```\n\n<!-- chunk: greet -->\n```go\nfmt.Println(\"𝄞\", whom)\n```\n"

// client is the editor's end of a conversation with the server.
type client struct {
    t        *testing.T
    to       io.Writer
    incoming chan *message
}

func connect(t *testing.T, gopls string) *client {
    toServer, fromClient := io.Pipe()
    fromServer, toClient := io.Pipe()
    go Serve(toServer, toClient, gopls)
    t.Cleanup(func() { fromClient.Close() })
    c := &client{t: t, to: fromClient, incoming: make(chan *message, 64)}
    go func() {
        r := bufio.NewReader(fromServer)
        for {
            m, err := read(r)
            if err != nil {
                return
            }
            c.incoming <- m
        }
    }()
    return c
}

func (c *client) send(method string, id int, params any) {
    raw, ok := params.(string)
    if !ok {
        b, _ := json.Marshal(params)
        raw = string(b)
    }
    body := fmt.Sprintf(`{"jsonrpc":"2.0","method":%q,"params":%s}`, method, raw)
    if id > 0 {
        body = fmt.Sprintf(`{"jsonrpc":"2.0","id":%d,"method":%q,"params":%s}`, id, method, raw)
    }
    fmt.Fprintf(c.to, "Content-Length: %d\r\n\r\n%s", len(body), body)
}

func (c *client) next() *message {
    select {
    case m := <-c.incoming:
        return m
    case <-time.After(60 * time.Second):
        c.t.Fatal("the server said nothing for a minute")
        return nil
    }
}

// diagnostics waits for a publication for the given version that satisfies ok.
func (c *client) diagnostics(version int, ok func(string) bool) string {
    for {
        m := c.next()
        if m.Method == "textDocument/publishDiagnostics" && strings.Contains(string(m.Params), fmt.Sprintf(`"version":%d`, version)) && ok(string(m.Params)) {
            return string(m.Params)
        }
    }
}

func TestDiagnosticsComeAndGo(t *testing.T) {
    c := connect(t, "off")
    c.send("initialize", 1, `{"capabilities":{}}`)
    if m := c.next(); !strings.Contains(fmt.Sprint(m.Result), "utf-16") {
        t.Fatalf("a client that offers nothing gets UTF-16: %v", m.Result)
    }
    uri := "file://" + t.TempDir() + "/hello.lit.md"
    c.send("textDocument/didOpen", 0, map[string]any{"textDocument": map[string]any{"uri": uri, "version": 1, "text": program}})
    got := c.diagnostics(1, func(p string) bool { return strings.Contains(p, "undefined: whom") })
    // Line 11, after fmt.Println(" (13 bytes), a clef that is 4 bytes but 2 UTF-16 units, and `", `.
    if !strings.Contains(got, `"start":{"line":11,"character":18}`) || !strings.Contains(got, `"end":{"line":11,"character":22}`) {
        t.Errorf("the error should underline whom, in UTF-16 units: %s", got)
    }

    c.send("textDocument/didChange", 0, map[string]any{
        "textDocument":   map[string]any{"uri": uri, "version": 2},
        "contentChanges": []any{map[string]any{"text": strings.Replace(program, "whom", `"world"`, 1)}},
    })
    c.diagnostics(2, func(p string) bool { return strings.Contains(p, `"diagnostics":[]`) })

    c.send("textDocument/formatting", 2, `{}`)
    if m := c.next(); m.Error == nil || m.Error.Code != -32601 {
        t.Errorf("an unknown request should be refused politely: %+v", m)
    }
}

Everything else gopls knows

internal/lsp/gopls.go

Showing errors is the bare minimum. A Go programmer’s editor also completes names, shows documentation, jumps to definitions, lists references and renames things. gopls, Go’s language server, already does all of that, and none of it should be written twice. There’s just one problem. gopls reads Go files, and you are in a Markdown file with the Go scattered around in the wrong order.

The source map runs in both directions, so litgo can stand in the middle. It starts its own gopls and shows it the tangled files as overlays (the protocol’s word for the text of a file open in an editor, which wins over what’s on disk). Nothing gets written. A question about a position in the document becomes the same question about a position in a tangled file, and every position in the answer gets translated back.

sequenceDiagram
    participant E as editor
    participant S as litgo lsp
    participant G as gopls
    E->>S: didChange: the whole buffer
    S->>G: didChange: pool.go, tangled, never written
    E->>S: definition? pool.lit.md:67:15
    S->>G: definition? pool.go:31:14
    G-->>S: pool.go:12:6
    S-->>E: pool.lit.md:40:6

The editor sees one server. Wherever the cursor is on Go, a .lit.md buffer is a Go buffer. And since nothing here is specific to Neovim, it works in any editor.

A server of our own

litgo looks for gopls on the $PATH first, then in the directory where go install puts it, since not everyone has that directory on their $PATH. No gopls? Then litgo is just the server from the last chapter.

// findGopls locates the gopls to run: the one named, or the one that can be
// found. "off" means none.
func findGopls(named string) string {
    if named == "off" {
        return ""
    }
    if named != "" {
        p, _ := exec.LookPath(named)
        return p
    }
    if p, err := exec.LookPath("gopls"); err == nil {
        return p
    }
    gopath := os.Getenv("GOPATH")
    if home, err := os.UserHomeDir(); gopath == "" && err == nil {
        gopath = filepath.Join(home, "go")
    }
    for _, dir := range []string{os.Getenv("GOBIN"), filepath.Join(gopath, "bin")} {
        if p, err := exec.LookPath(filepath.Join(dir, "gopls")); dir != "" && err == nil {
            return p
        }
    }
    return ""
}

From where gopls sits, litgo is the client. Requests are numbered, and pending holds what to do with each answer until it arrives. The editor might cancel a request in the meantime, so litgo also remembers the editor’s own number for it.

gopls adds two goroutines to the server, and zero locks. One writes to gopls’s standard input, the other reads its standard output and hands each message to the server’s loop over msgs. The loop does all the bookkeeping and runs every done function. So an answer from gopls is translated by the same goroutine that owns the documents and the overlays, and nobody has to share.

sequenceDiagram
    participant L as the loop
    participant W as writer
    participant G as gopls
    participant R as reader
    L->>W: out
    W->>G: standard input
    G->>R: standard output
    R->>L: msgs
// gopls is the Go language server, run as a child process. Like the rest of
// the server's state, it belongs to the loop in Serve.
type gopls struct {
    cmd     *exec.Cmd
    in      io.WriteCloser
    out     chan []byte   // to the goroutine that writes to gopls
    msgs    chan *message // from the goroutine that reads from it
    next    int
    pending map[int]call
    ids     map[string]int // the editor's id for a request in flight → ours
    dead    bool
    caps    map[string]any // what gopls said it can do
}

// call is a request in flight. done gets the answer, or nil if gopls died.
type call struct {
    done   func(*message)
    editor string
}

func (g *gopls) write(m *message) { g.out <- frame(m) }

func (g *gopls) notify(method string, params any) {
    raw, _ := json.Marshal(params)
    g.write(&message{Method: method, Params: raw})
}

// request asks gopls something on behalf of the editor's request id (nil if
// the question is litgo's own).
func (g *gopls) request(id *json.RawMessage, method string, params any, done func(*message)) {
    if g.dead {
        done(nil)
        return
    }
    raw, _ := json.Marshal(params)
    c := call{done: done}
    g.next++
    if id != nil {
        c.editor = string(*id)
        g.ids[c.editor] = g.next
    }
    g.pending[g.next] = c
    ours := json.RawMessage(strconv.Itoa(g.next))
    g.write(&message{ID: &ours, Method: method, Params: raw})
}

// cancel passes on the editor's loss of interest in one of its requests.
func (g *gopls) cancel(params json.RawMessage) {
    var p struct {
        ID json.RawMessage `json:"id"`
    }
    json.Unmarshal(params, &p)
    if n, ok := g.ids[string(p.ID)]; ok {
        g.notify("$/cancelRequest", map[string]any{"id": n})
    }
}

gopls sends three kinds of message. An answer goes to whoever is waiting for it. A request of its own (to register a capability, or read configuration) gets null back, which gopls takes to mean “use your defaults”. Of its notifications, litgo passes on only the error messages. The progress reports are about files you never see. The diagnostics would repeat the ones the type checker from the last chapter already placed, or worse, contradict them. And gopls’s one warning is that someone is editing a generated file. Yes, we know.

gopls is chatty, so the reader throws away the unwanted notifications before they reach the loop. When gopls hangs up, the reader closes msgs. The loop marks gopls as dead and tells everyone still waiting that no answer is coming.

// listen reads what gopls sends until it hangs up or the server stops.
func (g *gopls) listen(out io.Reader, quit <-chan struct{}) {
    defer close(g.msgs)
    defer g.cmd.Wait()
    r := bufio.NewReader(out)
    for {
        m, err := read(r)
        if err != nil {
            return
        }
        if m.ID == nil && m.Method != "window/showMessage" {
            continue
        }
        select {
        case g.msgs <- m:
        case <-quit:
            return
        }
    }
}

// heard is where messages from gopls arrive. Without a gopls it is nil, and
// a nil channel never delivers anything.
func (s *server) heard() <-chan *message {
    if s.gopls == nil || s.gopls.dead {
        return nil
    }
    return s.gopls.msgs
}

// hear handles one message from gopls.
func (g *gopls) hear(m *message, s *server) {
    switch {
    case m.Method == "" && m.ID != nil:
        n, _ := strconv.Atoi(string(*m.ID))
        c, ok := g.pending[n]
        delete(g.pending, n)
        delete(g.ids, c.editor)
        if ok {
            c.done(m)
        }
    case m.ID != nil:
        g.write(&message{ID: m.ID, Result: json.RawMessage("null")})
    case m.Method == "window/showMessage":
        var p struct {
            Type int `json:"type"`
        }
        if json.Unmarshal(m.Params, &p); p.Type == 1 {
            s.write(&message{Method: m.Method, Params: m.Params})
        }
    }
}

// hangUp is what happens when gopls is gone: nobody gets an answer any more.
func (g *gopls) hangUp() {
    g.dead = true
    orphans := g.pending
    g.pending = map[int]call{}
    for _, c := range orphans {
        c.done(nil)
    }
}

litgo starts gopls while the editor waits for its answer to initialize, because what litgo can promise the editor depends on what gopls promises litgo. It passes along what the editor said (the workspace, and what the editor can display, like snippets in completions and Markdown in hovers), minus the capabilities that would make gopls send requests of its own.

There’s a catch. The loop is the goroutine running start, so it can’t sit in its select while it waits for gopls. start reads from msgs itself until the answer shows up.

// start runs gopls and introduces litgo to it the way the editor introduced
// itself to litgo. On any failure s.gopls stays nil.
func (s *server) start(initialize json.RawMessage) {
    bin := findGopls(s.goplsBin)
    if bin == "" {
        return
    }
    cmd := exec.Command(bin)
    in, _ := cmd.StdinPipe()
    out, _ := cmd.StdoutPipe()
    if cmd.Start() != nil {
        return
    }
    g := &gopls{cmd: cmd, in: in, msgs: make(chan *message), pending: map[int]call{}, ids: map[string]int{}}
    g.out, _ = writer(in)
    go g.listen(out, s.quit)

    var p map[string]any
    if json.Unmarshal(initialize, &p); p == nil {
        p = map[string]any{}
    }
    theirs, _ := p["capabilities"].(map[string]any)
    p["capabilities"] = map[string]any{"textDocument": theirs["textDocument"]}
    var reply *message
    answered := false
    g.request(nil, "initialize", p, func(m *message) { reply, answered = m, true })
    timeout := time.After(30 * time.Second)
    for !answered {
        select {
        case m, ok := <-g.msgs:
            if ok {
                g.hear(m, s)
            } else {
                g.hangUp() // which answers the request, with nil
            }
        case <-timeout:
            answered = true
        }
    }
    if result, _ := reply.result().(map[string]any); result != nil {
        g.caps, _ = result["capabilities"].(map[string]any)
        g.notify("initialized", map[string]any{})
        s.gopls = g
        return
    }
    g.kill()
}

// kill stops gopls, and the goroutine that writes to it.
func (g *gopls) kill() {
    close(g.out)
    g.in.Close()
    g.cmd.Process.Kill()
}

// result is the result of a successful answer, and nil otherwise.
func (m *message) result() any {
    if m == nil || m.Error != nil {
        return nil
    }
    return m.Result
}

func (s *server) stop() {
    if g := s.gopls; g != nil {
        g.kill()
    }
}

Overlays

gopls has to see the latest text before anyone asks it a question. A completion request arrives right after the keystroke that triggered it, long before the delay diagnostics wait for. So tangling can’t wait either. Whoever needs the tangled files (the analysis, or a request) calls sync, and sync tangles at most once per version of the text. Both run on the loop, so they never trip over each other. A document with broken chunk structure still gets tangled as well as it can be. You’re in the middle of something, and you’d still like completions.

// overlay is a tangled file as gopls was last shown it.
type overlay struct {
    owner   string // URI of the document it is tangled from
    version int
    text    string
}

// sync tangles a document, unless that has been done for this version of
// its text, and shows gopls the result. It returns a nil result for a
// document that is not open.
func (s *server) sync(uri string) (*lit.Result, int) {
    d := s.docs[uri]
    if d == nil {
        return nil, -1
    }
    if d.res == nil || d.tangled != d.version {
        d.res, d.tangled = lit.Parse(path(uri), []byte(d.text)).Tangle(), d.version
        s.overlay(uri, d.res)
    }
    return d.res, d.version
}

// forget closes what a closed document tangled to.
func (s *server) forget(uri string) {
    s.overlay(uri, nil)
}

A document like this one tangles to dozens of files, and a keystroke changes one of them. litgo sends only that one.

// overlay brings gopls up to date with what a document tangles to: new
// files are opened, changed ones changed, and ones no longer written closed.
func (s *server) overlay(uri string, res *lit.Result) {
    g := s.gopls
    if g == nil {
        return
    }
    live := map[string]bool{}
    for _, o := range outputs(res) {
        live[o.Path] = true
        text := string(o.Bytes())
        ov := s.overlays[o.Path]
        switch {
        case ov == nil:
            s.overlays[o.Path] = &overlay{owner: uri, version: 1, text: text}
            g.notify("textDocument/didOpen", map[string]any{
                "textDocument": map[string]any{"uri": fileURI(o.Path), "languageId": "go", "version": 1, "text": text}})
        case ov.text != text:
            ov.owner, ov.text, ov.version = uri, text, ov.version+1
            g.notify("textDocument/didChange", map[string]any{
                "textDocument":   map[string]any{"uri": fileURI(o.Path), "version": ov.version},
                "contentChanges": []any{map[string]any{"text": text}}})
        }
    }
    for p, ov := range s.overlays {
        if ov.owner == uri && !live[p] {
            delete(s.overlays, p)
            g.notify("textDocument/didClose", map[string]any{
                "textDocument": map[string]any{"uri": fileURI(p)}})
        }
    }
}

// outputs lists the Go files of a result, which may be nil.
func outputs(res *lit.Result) []*lit.Output {
    var out []*lit.Output
    if res != nil {
        for _, o := range res.Files {
            if o.File.Lang == "go" {
                out = append(out, o)
            }
        }
    }
    return out
}

func fileURI(path string) string {
    return (&url.URL{Scheme: "file", Path: filepath.ToSlash(path)}).String()
}

When gopls names a file in an answer, it might be one of these overlays. output finds it and the document it came from, preferring the tangling that was current when the question was asked.

func (s *server) output(uri string, asked *lit.Result) (o *lit.Output, doc *lit.Doc, owner string) {
    p := filepath.Clean(path(uri))
    res := asked
    if ov := s.overlays[p]; ov != nil {
        owner = ov.owner
        if d := s.docs[owner]; d != nil && (asked == nil || asked.Doc.Path != path(owner)) {
            res = d.res
        }
    }
    if owner != "" {
        for _, o := range outputs(res) {
            if filepath.Clean(o.Path) == p {
                return o, res.Doc, owner
            }
        }
    }
    return nil, nil, ""
}

Questions

Here are the requests that ask about a position in a document, each with the capability that announces it. litgo offers the editor whatever gopls offers litgo, as long as it’s on this list. Four of them it always offers, because it can answer those itself when the position is on a chunk.

var positional = map[string]string{
    "textDocument/hover":             "hoverProvider",
    "textDocument/definition":        "definitionProvider",
    "textDocument/typeDefinition":    "typeDefinitionProvider",
    "textDocument/implementation":    "implementationProvider",
    "textDocument/references":        "referencesProvider",
    "textDocument/documentHighlight": "documentHighlightProvider",
    "textDocument/completion":        "completionProvider",
    "textDocument/signatureHelp":     "signatureHelpProvider",
    "textDocument/prepareRename":     "renameProvider",
    "textDocument/rename":            "renameProvider",
}

func (s *server) capabilities() map[string]any {
    c := map[string]any{}
    if s.gopls != nil {
        for _, name := range positional {
            if v, ok := s.gopls.caps[name]; ok {
                c[name] = v
            }
        }
        c["workspaceSymbolProvider"] = s.gopls.caps["workspaceSymbolProvider"]
    }
    c["hoverProvider"], c["definitionProvider"], c["referencesProvider"] = true, true, true
    c["renameProvider"] = map[string]any{"prepareProvider": true}
    return c
}
// proxy handles the requests that gopls has a say in, and reports whether
// the message was one.
func (s *server) proxy(m *message) bool {
    switch {
    case positional[m.Method] != "" && m.ID != nil:
        s.ask(m)
    case m.Method == "$/cancelRequest":
        if s.gopls != nil {
            s.gopls.cancel(m.Params)
        }
    case m.Method == "workspace/symbol" && m.ID != nil && s.gopls != nil:
        // No position to translate on the way in, but the symbols found
        // may well be in tangled files.
        var params any
        json.Unmarshal(m.Params, &params)
        s.gopls.request(m.ID, m.Method, params, func(reply *message) {
            s.relay(m, reply, &translation{s: s})
        })
    default:
        return false
    }
    return true
}

Locate translates the position in the question. A position in prose, in a Lua block, or on a line an exclude pattern drops isn’t anywhere in a Go file, so the answer is null. Every one of these requests allows null, and editors take it without a word.

// position is the protocol's: a line, and a column in the agreed units.
type position struct {
    Line      int `json:"line"`
    Character int `json:"character"`
}

func (s *server) ask(m *message) {
    var p struct {
        TextDocument struct {
            URI string `json:"uri"`
        } `json:"textDocument"`
        Position position `json:"position"`
        NewName  string   `json:"newName"`
    }
    json.Unmarshal(m.Params, &p)
    uri := p.TextDocument.URI
    res, _ := s.sync(uri)
    if res == nil || p.Position.Line < 0 || p.Position.Line >= len(res.Doc.Lines) {
        s.respond(m, nil)
        return
    }
    line := p.Position.Line
    col := offset(s.utf8, res.Doc.Lines[line], p.Position.Character)
    if mention := res.Doc.MentionAt(line, col); mention != nil {
        // <<answer a question about a chunk>>
        return
    }
    o, outLine, outCol, ok := res.Locate(line, col)
    if !ok || o.File.Lang != "go" || s.gopls == nil {
        s.respond(m, nil)
        return
    }
    var params map[string]any
    json.Unmarshal(m.Params, &params)
    params["textDocument"] = map[string]any{"uri": fileURI(o.Path)}
    params["position"] = position{outLine, units(false, o.Lines[outLine].Text, outCol)}
    s.gopls.request(m.ID, m.Method, params, func(reply *message) {
        s.relay(m, reply, &translation{s: s, res: res, out: o, uri: uri})
    })
}

// respond answers a request; a nil result is the protocol's null.
func (s *server) respond(to *message, result any) {
    if result == nil {
        result = json.RawMessage("null")
    }
    s.write(&message{ID: to.ID, Result: result})
}

gopls counts columns in UTF-16 code units, and nothing else. So when gopls is around, litgo stops agreeing to bytes with the editor. That way positions in files litgo knows nothing about (the standard library, a dependency) pass through untouched. For the documents and the tangled files, litgo has the text, so it converts the columns itself.

// offset converts a column in UTF-16 code units (or in bytes) to a byte
// offset in the line: the inverse of units.
func offset(utf8 bool, line string, n int) int {
    if utf8 {
        return min(max(n, 0), len(line))
    }
    for i, r := range line {
        if n <= 0 {
            return i
        }
        n--
        if r >= 0x10000 {
            n--
        }
    }
    return len(line)
}

Questions about chunks

To gopls, a chunk reference is just a comment, so it has nothing to say. But the questions still make sense, and litgo knows the answers. A reference is defined by the blocks of its chunk, and hovering shows those blocks. The references to a chunk are, well, its references, and renaming one is Rename. Ask for the definition of a definition and litgo answers with the uses, so one key takes you back and forth. None of this needs gopls.

⟨answer a question about a chunk⟩ ≡ used in the program
result, err := s.mention(m.Method, uri, res.Doc, mention, p.NewName)
if err != nil {
    s.write(&message{ID: m.ID, Error: &rpcError{requestFailed, err.Error()}})
} else {
    s.respond(m, result)
}
const requestFailed = -32803

func (s *server) mention(method, uri string, doc *lit.Doc, m *lit.Mention, newName string) (any, error) {
    span := func(line, col, end int) map[string]any {
        line = min(line, len(doc.Lines)-1)
        text := doc.Lines[line]
        return map[string]any{"start": position{line, s.column(text, col)}, "end": position{line, s.column(text, end)}}
    }
    c := m.Chunk
    var uses, blocks []any
    for _, r := range c.Refs {
        uses = append(uses, map[string]any{"uri": uri, "range": span(r.Line, r.Col, r.EndCol)})
    }
    for _, b := range c.Blocks {
        blocks = append(blocks, map[string]any{"uri": uri, "range": span(b.First(), 0, 0)})
    }
    switch method {
    case "textDocument/definition":
        if m.Ref {
            return blocks, nil
        }
        return uses, nil
    case "textDocument/references":
        return uses, nil
    case "textDocument/hover":
        if !m.Ref || len(c.Blocks) == 0 {
            return nil, nil
        }
        text := "⟨" + c.Name + "⟩ ≡\n"
        for _, b := range c.Blocks {
            text += "```" + b.Lang + "\n" + strings.Join(doc.Lines[b.First():b.Last()+1], "\n") + "\n```\n"
        }
        return map[string]any{"contents": map[string]any{"kind": "markdown", "value": text}, "range": span(m.Line, m.Col, m.EndCol)}, nil
    case "textDocument/prepareRename":
        return map[string]any{"range": span(m.Line, m.Col, m.EndCol), "placeholder": c.Name}, nil
    case "textDocument/rename":
        edits, err := doc.Rename(c.Name, newName)
        if err != nil {
            return nil, err
        }
        var changes []any
        for _, e := range edits { // Rename replaces lines one for one
            changes = append(changes, map[string]any{"range": span(e.Start, 0, len(doc.Lines[e.Start])), "newText": e.Lines[0]})
        }
        return map[string]any{"changes": map[string]any{uri: changes}}, nil
    }
    return nil, nil
}

Answers

The answers vary as much as the questions, but the positions inside them come in only a few shapes. So litgo translates the raw JSON, with no type for every kind of answer:

A chunk used twice is tangled twice, and gopls finds a reference in each copy. Both translate back to the same spot, so litgo drops the duplicates from lists.

// translation brings the answer to one question back to the document.
type translation struct {
    s   *server
    res *lit.Result // the tangling gopls was asked about,
    out *lit.Output // the file in it that the question was about,
    uri string      // and the document
}

func (t *translation) value(v any) any {
    switch v := v.(type) {
    case []any:
        if len(v) > 0 && t.out != nil && isEdit(asMap(v[0])) {
            return t.edits(t.out, t.res.Doc, v)
        }
        out, seen := []any{}, map[string]bool{}
        for _, e := range v {
            if e = t.value(e); e == nil {
                continue
            }
            if key := positioned(e); key == "" || !seen[key] {
                out = append(out, e)
                seen[key] = true
            }
        }
        return out
    case map[string]any:
        return t.object(v)
    }
    return v
}

func (t *translation) object(v map[string]any) any {
    if uri, ok := v["uri"].(string); ok && v["range"] != nil {
        if o, doc, owner := t.s.output(uri, t.res); o != nil {
            if v["range"], ok = t.span(o, doc, v["range"], false); !ok {
                return nil
            }
            v["uri"] = owner
        }
        return v
    }
    if uri, ok := v["targetUri"].(string); ok {
        if t.out != nil && v["originSelectionRange"] != nil {
            v["originSelectionRange"], _ = t.span(t.out, t.res.Doc, v["originSelectionRange"], false)
        }
        if o, doc, owner := t.s.output(uri, t.res); o != nil {
            for _, k := range []string{"targetRange", "targetSelectionRange"} {
                if v[k], ok = t.span(o, doc, v[k], false); !ok {
                    return nil
                }
            }
            v["targetUri"] = owner
        }
        return v
    }
    if t.out != nil && isEdit(v) { // on its own, as the textEdit of a completion
        if t.edit(t.out, t.res.Doc, v) {
            return v
        }
        return nil
    }
    if t.out != nil && isRange(v) {
        r, _ := t.span(t.out, t.res.Doc, v, false)
        return r
    }
    for k, e := range v {
        if e = t.value(e); e == nil {
            delete(v, k) // in the protocol, null and absent are the same
        } else {
            v[k] = e
        }
    }
    return v
}

func isRange(v map[string]any) bool {
    _, start := v["start"].(map[string]any)
    _, end := v["end"].(map[string]any)
    return start && end
}

func isEdit(v map[string]any) bool {
    _, ok := v["newText"].(string)
    return ok
}

func asMap(v any) map[string]any {
    m, _ := v.(map[string]any)
    return m
}

func asList(v any) []any {
    l, _ := v.([]any)
    return l
}

// positioned gives a location, a highlight or an edit a key to be compared
// by, and anything else none.
func positioned(v any) string {
    if m, ok := v.(map[string]any); ok && (m["range"] != nil || m["targetUri"] != nil) {
        b, _ := json.Marshal(m)
        return string(b)
    }
    return ""
}

A range is translated one end at a time, exactly if possible. A range that’s only for looking at can fall back on Map’s approximation: the definition of a name in a generated import, say, is close enough to the imports: directive. A range to replace gets no such mercy. It also needs the same shape in the document as in the tangled file, and it doesn’t have it if it starts in a host line and ends in a chunk spliced into that line.

// span translates a range in a tangled file to the document it came from.
func (t *translation) span(o *lit.Output, doc *lit.Doc, v any, edit bool) (any, bool) {
    r, _ := v.(map[string]any)
    l1, c1, ok1 := outPosition(o, r["start"])
    l2, c2, ok2 := outPosition(o, r["end"])
    if !ok1 || !ok2 && edit {
        return v, false
    }
    if !ok2 {
        l2, c2 = l1, c1
    }
    sl, sc, ok1 := o.Exact(l1, c1, false)
    el, ec, ok2 := o.Exact(l2, c2, l1 != l2 || c1 != c2)
    if edit && !(ok1 && ok2 && el-sl == l2-l1 && (l1 != l2 || ec-sc == c2-c1)) {
        return v, false
    }
    if !ok1 {
        if sl, sc, _, ok1 = o.Map(l1, c1); !ok1 {
            return v, false
        }
    }
    if !ok2 {
        el, ec, _, ok2 = o.Map(l2, c2)
    }
    if !ok2 || el < sl || el == sl && ec < sc {
        el, ec = sl, sc
    }
    return map[string]any{
        "start": position{sl, units(t.s.utf8, doc.Lines[sl], sc)},
        "end":   position{el, units(t.s.utf8, doc.Lines[el], ec)},
    }, true
}

// outPosition decodes a position in a tangled file, in bytes.
func outPosition(o *lit.Output, v any) (line, col int, ok bool) {
    p, _ := v.(map[string]any)
    l, ok1 := p["line"].(float64)
    c, ok2 := p["character"].(float64)
    if !ok1 || !ok2 || l < 0 || int(l) >= len(o.Lines) {
        return 0, 0, false
    }
    return int(l), offset(false, o.Lines[int(l)].Text, int(c)), true
}

An edit that can’t be translated gets dropped, with one exception. Complete strings.ToUpper in a file that doesn’t import strings, and gopls sends edits to the import ( block, which litgo generated from the imports: directive. As they stand the edits are useless. gopls sends the smallest difference it can find, and between "fmt" and "fmt", newline, "strings" that’s a few characters in the middle of a line. But what the edits are for is easy to find out. litgo carries them out on the tangled file and looks at what the file imports now that it didn’t before. AddImport knows how to add that to a document.

// edit translates a text edit in place, if it has a place in the document.
func (t *translation) edit(o *lit.Output, doc *lit.Doc, v map[string]any) bool {
    ok := true
    for _, k := range []string{"range", "insert", "replace"} { // a completion may offer both an insertion and a replacement
        if r, has := v[k]; has && ok {
            if l, _, _ := outPosition(o, asMap(r)["start"]); l < o.Header {
                return false
            }
            v[k], ok = t.span(o, doc, r, true)
        }
    }
    return ok
}

// edits translates a list of text edits. Those to the generated header
// become edits to the imports directive.
func (t *translation) edits(o *lit.Output, doc *lit.Doc, list []any) []any {
    out, header := []any{}, []map[string]any{}
    for _, e := range list {
        m := asMap(e)
        if l, _, ok := outPosition(o, asMap(m["range"])["start"]); ok && l < o.Header {
            header = append(header, m)
        } else if t.edit(o, doc, m) {
            out = append(out, m)
        }
    }
    if len(header) == 0 {
        return out
    }
    // <<carry out the edits to the header>>
    file, _ := parser.ParseFile(token.NewFileSet(), "", text, parser.ImportsOnly)
    if file == nil {
        return out
    }
    for _, spec := range file.Imports {
        alias, path := "", strings.Trim(spec.Path.Value, "\"`")
        if spec.Name != nil {
            alias = spec.Name.Name
        }
        known := false
        for _, imp := range o.File.Imports {
            known = known || imp.Path == path
        }
        if line, col, entry, ok := doc.AddImport(o.File, alias, path); ok && !known {
            at := position{line, units(t.s.utf8, doc.Lines[min(line, len(doc.Lines)-1)], col)}
            out = append(out, map[string]any{"range": map[string]any{"start": at, "end": at}, "newText": entry})
        }
    }
    return out
}

Every edit in a list refers to the text as it was before any of them, so they get carried out from the bottom up.

⟨carry out the edits to the header⟩ ≡ used in the program
text := string(o.Bytes())
starts := []int{0}
for _, l := range o.Lines {
    starts = append(starts, starts[len(starts)-1]+len(l.Text)+1)
}
at := func(v any) int {
    l, c, ok := outPosition(o, v)
    if !ok {
        return -1
    }
    return starts[l] + c
}
sort.SliceStable(header, func(i, j int) bool {
    return at(asMap(header[i]["range"])["start"]) > at(asMap(header[j]["range"])["start"])
})
for _, m := range header {
    r := asMap(m["range"])
    if from, to := at(r["start"]), at(r["end"]); from >= 0 && to >= from {
        text = text[:from] + m["newText"].(string) + text[to:]
    }
}

Renaming is the one answer made entirely of edits, across any number of files. It’s also the one where dropping an edit is not an option. Half a rename is a program that doesn’t compile. So if any part can’t be done, litgo refuses the whole thing. That includes gopls wanting to change a tangled file whose document isn’t open. The change would vanish at the next tangle. The right move is to open that document and rename there.

The edits come back grouped by tangled file and go out grouped by document. litgo strips the version numbers gopls put on them, because they are versions of overlays the editor has never heard of.

// workspaceEdit translates the answer to a rename.
func (t *translation) workspaceEdit(v any) (any, error) {
    w, _ := v.(map[string]any)
    if w == nil {
        return nil, nil
    }
    changes := map[string][]any{}
    seen := map[string]bool{}
    add := func(uri string, edits []any) error {
        o, doc, owner := t.s.output(uri, t.res)
        if o == nil {
            if b, err := os.ReadFile(path(uri)); err == nil && lit.Generated(b) {
                return fmt.Errorf("%s is tangled from a document that is not open: open it, and rename there", filepath.Base(path(uri)))
            }
            changes[uri] = append(changes[uri], edits...)
            return nil
        }
        for _, e := range edits {
            m := asMap(e)
            at, _ := json.Marshal(m["range"])
            if !t.edit(o, doc, m) {
                return fmt.Errorf("this would change text that litgo generates, in %s at %s", filepath.Base(o.Path), at)
            }
            if key := positioned(m); !seen[owner+key] {
                seen[owner+key] = true
                changes[owner] = append(changes[owner], m)
            }
        }
        return nil
    }
    for uri, edits := range asMap(w["changes"]) {
        if err := add(uri, asList(edits)); err != nil {
            return nil, err
        }
    }
    for _, c := range asList(w["documentChanges"]) {
        change := asMap(c)
        if change["kind"] != nil {
            return nil, fmt.Errorf("this would %s a file, which a document's file directives decide", change["kind"])
        }
        uri, _ := asMap(change["textDocument"])["uri"].(string)
        if err := add(uri, asList(change["edits"])); err != nil {
            return nil, err
        }
    }
    return map[string]any{"changes": changes}, nil
}
// relay passes gopls's answer to a request on to the editor.
func (s *server) relay(to *message, reply *message, t *translation) {
    switch {
    case reply != nil && reply.Error != nil:
        s.write(&message{ID: to.ID, Error: reply.Error})
    case to.Method == "textDocument/rename":
        edit, err := t.workspaceEdit(reply.result())
        if err != nil {
            s.write(&message{ID: to.ID, Error: &rpcError{requestFailed, err.Error()}})
            return
        }
        s.respond(to, edit)
    default:
        s.respond(to, t.value(reply.result()))
    }
}
internal/lsp/gopls_test.go

The test needs gopls and skips itself without it. Its document has a function in one block, a call to it in a chunk, and a treble clef in front of the call (two UTF-16 units, four bytes, maximum trouble). Every position in the test is translated twice on the way in and twice on the way out.

const greeter = "<!-- package: main -->\n<!-- imports: fmt -->\n\n```go\nfunc greet(whom string) string {\n\treturn \"hello, \" + whom\n}\n\nfunc main() {\n\t// <<say it>>\n}\n```\n\n<!-- chunk: say it -->\n```go\nfmt.Println(\"𝄞\", greet(\"world\"))\n```\n"

// ask sends a request about a position and returns the result as JSON.
func (c *client) ask(id int, method, uri string, line, character int, more map[string]any) string {
    params := map[string]any{"textDocument": map[string]any{"uri": uri}, "position": map[string]any{"line": line, "character": character}}
    for k, v := range more {
        params[k] = v
    }
    c.send("textDocument/"+method, id, params)
    return c.result(id)
}

// result waits for the answer to a request.
func (c *client) result(id int) string {
    for {
        if m := c.next(); m.ID != nil && string(*m.ID) == strconv.Itoa(id) {
            if m.Error != nil {
                return "error: " + m.Error.Message
            }
            var sb strings.Builder
            enc := json.NewEncoder(&sb)
            enc.SetEscapeHTML(false)
            enc.Encode(m.Result)
            return strings.TrimSpace(sb.String())
        }
    }
}

// start spells the start of a range the way encoding/json does: keys sorted.
func start(line, character int) string {
    return fmt.Sprintf(`"start":{"character":%d,"line":%d}`, character, line)
}

func TestGoplsThroughTheDocument(t *testing.T) {
    if findGopls("") == "" {
        t.Skip("no gopls")
    }
    dir := t.TempDir()
    os.WriteFile(dir+"/go.mod", []byte("module example.com/greeter\n\ngo 1.22\n"), 0o644)
    uri := "file://" + dir + "/greeter.lit.md"
    c := connect(t, "")
    c.send("initialize", 1, map[string]any{"rootUri": "file://" + dir, "capabilities": map[string]any{
        "general": map[string]any{"positionEncodings": []string{"utf-8", "utf-16"}}}})
    if got := c.result(1); !strings.Contains(got, "completionProvider") || !strings.Contains(got, `"utf-16"`) {
        t.Fatalf("with gopls behind it the server completes, and counts as gopls does: %s", got)
    }
    c.send("initialized", 0, `{}`)
    c.send("textDocument/didOpen", 0, map[string]any{"textDocument": map[string]any{"uri": uri, "version": 1, "text": greeter}})

    // Println, in the chunk. gopls may still be loading the first time.
    hover := ""
    for i := 0; i < 50 && !strings.Contains(hover, "func fmt.Println"); i++ {
        if hover = c.ask(10+i, "hover", uri, 15, 5, nil); hover == "null" {
            time.Sleep(200 * time.Millisecond)
        }
    }
    if !strings.Contains(hover, "func fmt.Println") || !strings.Contains(hover, start(15, 4)) {
        t.Errorf("hover in a chunk: %s", hover)
    }
    // greet, after the clef: from the chunk to the block that defines it.
    if got := c.ask(2, "definition", uri, 15, 19, nil); !strings.Contains(got, uri) || !strings.Contains(got, start(4, 5)) {
        t.Errorf("definition across blocks: %s", got)
    }
    if got := c.ask(3, "references", uri, 4, 6, map[string]any{"context": map[string]any{"includeDeclaration": true}}); strings.Count(got, uri) != 2 || !strings.Contains(got, start(15, 18)) {
        t.Errorf("references, in UTF-16 units: %s", got)
    }
    if got := c.ask(4, "rename", uri, 4, 6, map[string]any{"newName": "hail"}); strings.Count(got, `"newText":"hail"`) != 2 ||
        !strings.Contains(got, start(15, 18)) || strings.Contains(got, "greeter.go") {
        t.Errorf("rename edits the document, not the tangled file: %s", got)
    }
    if got := c.ask(5, "hover", uri, 0, 3, nil); got != "null" {
        t.Errorf("prose is not Go: %s", got)
    }

    // The chunk reference is litgo's own business.
    if got := c.ask(6, "definition", uri, 9, 6, nil); !strings.Contains(got, start(15, 0)) {
        t.Errorf("a reference is defined by its chunk: %s", got)
    }
    if got := c.ask(7, "hover", uri, 9, 6, nil); !strings.Contains(got, "⟨say it⟩") || !strings.Contains(got, "greet(") {
        t.Errorf("hovering over a reference shows the chunk: %s", got)
    }
    if got := c.ask(8, "rename", uri, 13, 14, map[string]any{"newName": "speak"}); !strings.Contains(got, "<!-- chunk: speak -->") || !strings.Contains(got, "// <<speak>>") {
        t.Errorf("renaming a chunk: %s", got)
    }

    // Completing from a package that is not imported adds it to the directive.
    c.send("textDocument/didChange", 0, map[string]any{
        "textDocument":   map[string]any{"uri": uri, "version": 2},
        "contentChanges": []any{map[string]any{"text": strings.Replace(greeter, `greet("world")`, "strings.", 1)}},
    })
    got := c.ask(9, "completion", uri, 15, 26, nil)
    if !strings.Contains(got, `"label":"ToUpper"`) {
        t.Fatalf("completion: %.300s", got)
    }
    if !strings.Contains(got, `"newText":", strings"`) || !strings.Contains(got, start(1, 17)) {
        i := strings.Index(got, `"label":"ToUpper"`)
        t.Errorf("the import belongs in the imports directive: %.600s", got[max(i-300, 0):])
    }
}

The Neovim plugin

The plugin lives at the root of this repository, where plugin managers look for plugin/ and lua/. It is a client of the protocol above, so the two change together. With lazy.nvim:

{
  "tlehman/litgo",
  build = "go build -o bin/litgo .",
  event = { "BufReadPre *.lit.md", "BufNewFile *.lit.md" },
  opts = {},
}

The plugin is thin on purpose. Everything that knows about the format lives in the binary. The Lua just turns answers into extmarks, edits and diagnostics. Another editor would need the same four calls and nothing else.

Command Key
(nothing) Chunk, syntax and type errors show up under the offending line as you type, from litgo lsp. So do failed proofs of //@ annotations, in violet. The annotations themselves are violet too, and their formulas are typeset like the math in the prose: lo*lo <= n ^ n < hi*hi reads lo^2 \leq n \land n < hi^2 until the cursor lands on the line.
(what Neovim has for any language server) K, grr, grn, <C-x><C-o>, <C-s> In a Go block the buffer acts like a Go buffer: gopls’s completion, documentation, references, rename and signature help, all through litgo lsp.
:TangleCompileAndRun [args] \r Saves, tangles, compiles and runs. Panics and failing tests land in the Markdown too, and the cursor jumps to the first error.
:'<,'>Untangle [name] \u Moves the selection into a chunk at the bottom and leaves a reference.
:RenameChunk [name] \n Renames the chunk under the cursor, everywhere.
:Prove \p Saves and checks the //@ annotations, without compiling or running.
:Tangle[!], :Weave [html] \t, \w :Tangle writes the sources. :Weave writes and opens the PDF, or the web page.
gd, K On a chunk, gd goes from a reference to the definition and from a definition to the uses, and K previews the chunk. On Go they do what they do in Go.
:LitWatch, :LitStop, :LitClear, :LitRender Run on save, kill the program, clear diagnostics, toggle rendering.
plugin/litgo.lua
-- Loaded automatically when this directory is on the runtimepath. Calling
-- require("litgo").setup({...}) yourself (before or after) overrides options.
if vim.g.loaded_litgo then
  return
end

vim.g.loaded_litgo = true

if vim.fn.has("nvim-0.11") == 0 then
  vim.notify("litgo needs Neovim 0.11 or newer (inline virtual text, conceal_lines)", vim.log.levels.WARN)
  return
end

require("litgo").setup()

Setup

lua/litgo/init.lua
-- litgo.nvim: a tight loop for literate Go in .lit.md files.
--
-- The plugin is deliberately thin. Everything that knows about the format
-- lives in the `litgo` binary; Lua asks it questions (document on stdin, JSON
-- back) and turns the answers into extmarks, edits and diagnostics.
local M = {}

M.config = {
  bin = nil, -- path to the litgo binary; found on $PATH when nil
  render = {
    enabled = true,
    math = true,
    mermaid = true,
    tables = true, -- line up the columns of Markdown tables
    proofs = true, -- typeset the formulas of //@ annotations: ^ as ∧, <= as ≤, Forall as ∀
    chunks = true, -- show `// <<name>>` as ⟨name⟩ and chunk directives as ⟨name⟩ ≡
    conceal_source = true, -- hide diagram/display-math source while the cursor is elsewhere
    upright = false, -- true: keep math letters upright instead of 𝑖𝑡𝑎𝑙𝑖𝑐
    indent = 4,
    debounce_ms = 150,
  },
  run = {
    vet = false, -- also report `go vet` findings as warnings
    timeout = "60s", -- kill runaway programs; "" to disable
    jump = true, -- move the cursor to the first error
    focus = true, -- put the cursor in the output panel when the run ends, so q closes it
    on_save = false, -- :TangleCompileAndRun after every write (toggle with :LitWatch)
    panel_height = 12,
  },
  lsp = {
    enabled = true, -- run `litgo lsp`: chunk, syntax, type and proof errors as you type
    virtual_lines = true, -- show them under the offending line rather than beside it
    gopls = true, -- completion, hover, definitions, references, rename in Go blocks; or a path, or false
    settings = nil, -- gopls settings, e.g. { buildFlags = { "-tags=integration" } }
    completion = "auto", -- "auto": Neovim's own completion popup as you type, unless a
    -- completion plugin is loaded. true forces it on, false off.
  },
  weave = { open = true },
  scaffold = true, -- fill a new, empty .lit.md with a program that already runs
  untangle = { jump = false }, -- true: go to the new chunk to document it right away
  keymaps = {
    run = "<localleader>r",
    stop = "<localleader>s",
    tangle = "<localleader>t",
    weave = "<localleader>w",
    prove = "<localleader>p",
    untangle = "<localleader>u", -- visual mode
    rename = "<localleader>n",
    toggle_render = "<localleader>v",
    goto_chunk = "gd",
    hover = "K",
  },
}

The plugin looks for the binary in its own checkout first. That’s where a plugin manager’s build step puts it, and a binary from there is sure to match the Lua.

local function plugin_root()
  local src = debug.getinfo(1, "S").source:sub(2)
  return vim.fn.fnamemodify(src, ":p:h:h:h")
end

local warned = false

--- Locate the litgo binary: explicit config, a build inside the plugin's own
--- checkout (what a plugin manager's build step produces), then $PATH.
function M.bin()
  if M._bin then
    return M._bin
  end
  local candidates = { plugin_root() .. "/bin/litgo", "litgo" }
  if M.config.bin then
    table.insert(candidates, 1, vim.fn.expand(M.config.bin))
  end
  for _, c in ipairs(candidates) do
    if c and vim.fn.executable(c) == 1 then
      M._bin = c
      return c
    end
  end
  if not warned then
    warned = true
    vim.notify(
      "litgo: binary not found. Run `go build -o bin/litgo .` in " .. plugin_root() .. ", or set `bin` in setup().",
      vim.log.levels.ERROR
    )
  end
end

To the plugin, the protocol has two halves: ask a question about the buffer, then apply the edits that come back. Several nvim_buf_set_lines calls made from one command count as a single undo step.

--- Run litgo synchronously with the buffer on stdin; returns decoded JSON.
function M.query(buf, args)
  local bin = M.bin()
  if not bin then
    return nil, "litgo binary not found"
  end
  local cmd = vim.list_extend({ bin }, args)
  local text = table.concat(vim.api.nvim_buf_get_lines(buf, 0, -1, false), "\n") .. "\n"
  local res = vim.system(cmd, { stdin = text, text = true }):wait(5000)
  local ok, decoded = pcall(vim.json.decode, res.stdout or "")
  if not ok or type(decoded) ~= "table" then
    return nil, (res.stderr ~= "" and res.stderr) or "litgo returned no JSON"
  end
  if decoded.error then
    return nil, decoded.error
  end
  return decoded
end

--- Apply bottom-up line edits as a single undo step.
function M.apply_edits(buf, edits)
  for _, e in ipairs(edits) do
    vim.api.nvim_buf_set_lines(buf, e.start, e["end"], false, e.lines)
  end
end

function M.is_lit(buf)
  return vim.api.nvim_buf_get_name(buf):match("%.lit%.md$") ~= nil
end

local function define_highlights()
  local links = {
    LitgoMath = "Special",
    LitgoDiagram = "Function",
    LitgoTableRule = "Comment",
    LitgoChunk = "Title",
    LitgoFile = "Directory",
    LitgoInfo = "Comment",
    LitgoError = "DiagnosticError",
    LitgoOk = "DiagnosticOk",
  }
  for name, target in pairs(links) do
    vim.api.nvim_set_hl(0, name, { link = target, default = true })
  end
  -- <<the colour of a proof>>
end

Everything about proofs is violet: the //@ annotations in the code, and whatever the verifier has to say about them. No colour scheme has a group for that, and the whole point is that it isn’t one of the others. A red line under code says the compiler won’t take it. A yellow one says go vet frowns. A violet one is news of a different kind: this compiles, it runs, and nobody has shown that it does what it promises. The groups are defined with default, so a colour scheme (or a user with strong opinions about violet) wins.

⟨the colour of a proof⟩ ≡ used in the program
local dark = vim.o.background ~= "light"
local violet = dark and "#c792ea" or "#7c3aed"
local faded = dark and "#a58fc4" or "#8b6fc0"
for name, spec in pairs({
  LitgoProofKeyword = { fg = violet, bold = true },
  LitgoProof = { fg = faded },
  LitgoProofError = { fg = violet },
  LitgoProofWarn = { fg = faded, italic = true },
  LitgoProofUnderline = { undercurl = true, sp = violet },
}) do
  spec.default = true
  vim.api.nvim_set_hl(0, name, spec)
end

A completion plugin offers the same sources everywhere in the buffer. In a Go block that means Markdown snippets and words from the prose, mixed in with what gopls has to say. The plugin can’t reach into someone else’s configuration, so it answers the question the configuration needs: is this line inside a Go block? Counting fences from the top is enough, and cheap, because a completion menu only asks about the line the cursor is on.

--- Is a line (0-based; the cursor's by default) inside a Go block?
function M.in_go_block(buf, line)
  buf = buf or 0
  if not M.is_lit(buf) then
    return false
  end
  line = line or (vim.api.nvim_win_get_cursor(0)[1] - 1)
  local lang, fence = nil, nil
  for _, l in ipairs(vim.api.nvim_buf_get_lines(buf, 0, line, false)) do
    local marks, rest = l:match("^%s*(``+`)%s*(.*)$")
    if marks then
      if not fence then
        fence, lang = marks, rest:match("^[%w_+-]*")
      elseif #marks >= #fence then
        fence, lang = nil, nil
      end
    end
  end
  return lang == "go"
end

A new file starts out as an empty buffer, and an empty buffer is not a literate program: there are no Go blocks, so there is nothing to tangle, and the language server says so on line 1 before a word has been typed. Rather than greet a new file with a warning, the plugin fills it with the smallest program that runs. The three directives at the top say which tangler to use, which package the code goes in and what it imports; the title comes from the file name, so foo.lit.md is # foo. Only an empty buffer gets this, whether the file is new or was made with touch, so nothing is ever overwritten.

--- Fill an empty .lit.md buffer with a program that runs.
local function scaffold(buf)
  if vim.api.nvim_buf_line_count(buf) > 1 or vim.api.nvim_buf_get_lines(buf, 0, 1, false)[1] ~= "" then
    return
  end
  local name = vim.api.nvim_buf_get_name(buf)
  local title = vim.fn.fnamemodify(name, ":t"):gsub("%.lit%.md$", "")
  vim.api.nvim_buf_set_lines(buf, 0, -1, false, {
    "<!-- tangler: go -->",
    "<!-- package: main -->",
    "<!-- imports: fmt -->",
    "",
    "# " .. title,
    "",
    "```go",
    "func main() {",
    '\tfmt.Println("Hello from ' .. title .. '!")',
    "}",
    "```",
  })
end

The filetype stays markdown, so treesitter highlighting and everything else that works for Markdown keeps working. The commands are local to the buffer.

--- Set up one .lit.md buffer: commands, keymaps, rendering.
function M.attach(buf)
  if vim.b[buf].litgo_attached or not M.is_lit(buf) then
    return
  end
  vim.b[buf].litgo_attached = true
  if M.config.scaffold then
    scaffold(buf)
  end

  local run = require("litgo.run")
  local chunks = require("litgo.chunks")
  local render = require("litgo.render")

  local function command(name, fn, opts)
    vim.api.nvim_buf_create_user_command(buf, name, fn, opts or {})
  end
  command("Tangle", function(o)
    run.tangle(buf, o.bang)
  end, { bang = true, desc = "Write the Go source (! overwrites files litgo did not generate)" })
  command("Weave", function(o)
    run.weave(buf, o.args)
  end, { nargs = "?", desc = "Write and open the woven PDF (or :Weave html)" })
  command("TangleCompileAndRun", function(o)
    run.run(buf, o.fargs)
  end, { nargs = "*", desc = "Tangle, compile and run; errors appear inline" })
  command("Prove", function()
    run.prove(buf)
  end, { desc = "Check the //@ annotations: contracts, invariants, termination" })
  command("LitStop", function()
    run.stop()
  end, { desc = "Kill the running program" })
  command("LitWatch", function()
    M.config.run.on_save = not M.config.run.on_save
    vim.notify("litgo: run on save " .. (M.config.run.on_save and "on" or "off"))
  end, { desc = "Toggle :TangleCompileAndRun on every write" })
  command("LitClear", function()
    run.clear(buf)
  end, { desc = "Clear compile and run diagnostics" })
  command("Untangle", function(o)
    chunks.untangle(buf, o)
  end, { range = true, nargs = "*", desc = "Move the selection into a named chunk at the bottom" })
  command("RenameChunk", function(o)
    chunks.rename(buf, o.args)
  end, { nargs = "*", desc = "Rename the chunk under the cursor everywhere" })
  command("LitRender", function()
    render.toggle(buf)
  end, { desc = "Toggle math, diagram and chunk rendering" })

  local keys = M.config.keymaps or {}
  local function map(mode, lhs, rhs, desc)
    if lhs and lhs ~= "" then
      vim.keymap.set(mode, lhs, rhs, { buffer = buf, silent = true, desc = "litgo: " .. desc })
    end
  end
  map("n", keys.run, "<cmd>TangleCompileAndRun<cr>", "tangle, compile and run")
  map("n", keys.stop, "<cmd>LitStop<cr>", "stop the program")
  map("n", keys.tangle, "<cmd>Tangle<cr>", "tangle")
  map("n", keys.weave, "<cmd>Weave<cr>", "weave")
  map("n", keys.prove, "<cmd>Prove<cr>", "prove the annotations")
  map("x", keys.untangle, ":Untangle<cr>", "untangle selection into a chunk")
  map("n", keys.rename, "<cmd>RenameChunk<cr>", "rename chunk")
  map("n", keys.toggle_render, "<cmd>LitRender<cr>", "toggle rendering")
  map("n", keys.goto_chunk, chunks.goto_chunk, "go to chunk definition / uses")
  map("n", keys.hover, chunks.hover, "preview chunk")

  vim.api.nvim_create_autocmd("BufWritePost", {
    buffer = buf,
    callback = function()
      if M.config.run.on_save then
        run.run(buf, {})
      end
    end,
  })

  run.attach(buf)
  render.attach(buf)
  require("litgo.lsp").start(buf)
end

function M.setup(opts)
  M.config = vim.tbl_deep_extend("force", M.config, opts or {})
  M._bin = nil
  define_highlights()
  require("litgo.proof").setup()
  local group = vim.api.nvim_create_augroup("litgo", { clear = true })
  vim.api.nvim_create_autocmd({ "BufReadPost", "BufNewFile", "BufWinEnter" }, {
    group = group,
    pattern = "*.lit.md",
    callback = function(ev)
      M.attach(ev.buf)
    end,
  })
  vim.api.nvim_create_autocmd("ColorScheme", { group = group, callback = define_highlights })
  for _, buf in ipairs(vim.api.nvim_list_bufs()) do
    if vim.api.nvim_buf_is_loaded(buf) then
      M.attach(buf)
    end
  end
end

return M

Rendering

lua/litgo/render.lua
-- In-buffer rendering: LaTeX as Unicode, Mermaid as box drawings, and chunk
-- syntax as ⟨names⟩. Whatever the cursor touches falls back to its source,
-- with block renderings kept underneath as a live preview.
local M = {}

local ns = vim.api.nvim_create_namespace("litgo-render")

local check_ns = vim.api.nvim_create_namespace("litgo-check")

M.check_ns = check_ns

-- bufnr -> { items, chunks, enabled, dirty, timer, cursor_row }
local state = {}

local function litgo()
  return require("litgo")
end

function M.get(buf)
  return state[buf]
end

local function virt_lines(lines, hl, indent)
  local pad = string.rep(" ", indent)
  local out = {}
  for _, l in ipairs(lines) do
    out[#out + 1] = { { pad .. l, hl } }
  end
  return out
end

local function mark(buf, row, col, opts)
  opts.strict = false
  pcall(vim.api.nvim_buf_set_extmark, buf, ns, row, col, opts)
end

local function wanted(item, cfg)
  if item.kind == "math" or item.kind == "math-block" then
    return cfg.math
  elseif item.kind == "mermaid" then
    return cfg.mermaid
  elseif item.kind == "table" then
    return cfg.tables
  elseif item.kind == "vego-math" then
    return cfg.proofs
  end
  return cfg.chunks
end

Everything is drawn with extmarks, and whatever the cursor is on is left undrawn. The source is always one keystroke away from its rendering.

--- Redraw every extmark. Cheap enough (one pass over the items) to run on
--- each cursor-line change, which keeps the reveal logic trivial.
function M.draw(buf)
  local st = state[buf]
  if not st or not vim.api.nvim_buf_is_valid(buf) then
    return
  end
  vim.api.nvim_buf_clear_namespace(buf, ns, 0, -1)
  if not st.enabled or not st.items then
    return
  end
  local cfg = litgo().config.render
  local cursor = -1
  if vim.api.nvim_get_current_buf() == buf then
    cursor = vim.api.nvim_win_get_cursor(0)[1] - 1
  end
  st.cursor_row = cursor
  local last = vim.api.nvim_buf_line_count(buf) - 1

  for _, it in ipairs(st.items) do
    if it.kind == "vego" then
      -- An annotation keeps its colour under the cursor too: nothing is
      -- hidden, so there is nothing to reveal.
      local key_end = it.col + 3
      if it.name and it.name ~= "" then
        local line = vim.api.nvim_buf_get_lines(buf, it.line, it.line + 1, false)[1] or ""
        local _, e = line:find(it.name, it.col + 4, true)
        key_end = e or key_end
      end
      mark(buf, it.line, it.col, { end_col = key_end, hl_group = "LitgoProofKeyword", priority = 210 })
      mark(buf, it.line, key_end, { end_col = it.end_col, hl_group = "LitgoProof", priority = 210 })
    elseif wanted(it, cfg) then
      local under_cursor = cursor >= it.line and cursor <= it.end_line
      if it.lines then
        -- <<draw a diagram or a displayed formula>>
      elseif it.table then
        -- <<line up the columns of a table>>
      else
        -- <<draw inline math or a chunk name>>
      end
    end
  end
end

A rendered block replaces its source. The source lines are concealed, and the drawing hangs off a neighbouring line as virtual lines. Move the cursor inside the block and the source comes back, with the drawing underneath as a live preview.

⟨draw a diagram or a displayed formula⟩ ≡ used in the program
local hl = it.kind == "mermaid" and "LitgoDiagram" or "LitgoMath"
if it.error then
  hl = "LitgoError"
end
local vl = virt_lines(it.lines, hl, cfg.indent)
if under_cursor or not cfg.conceal_source then
  -- Editing: keep the source visible and preview underneath. Anchor to
  -- the line after the block: treesitter conceals fence lines, and
  -- virtual lines hanging off a concealed line disappear with it.
  if it.end_line < last then
    mark(buf, it.end_line + 1, 0, { virt_lines = vl, virt_lines_above = true })
  else
    mark(buf, it.end_line, 0, { virt_lines = vl })
  end
elseif it.end_line < last then
  mark(buf, it.line, 0, { end_row = it.end_line, conceal_lines = "" })
  mark(buf, it.end_line + 1, 0, { virt_lines = vl, virt_lines_above = true })
elseif it.line > 0 then
  mark(buf, it.line, 0, { end_row = it.end_line, conceal_lines = "" })
  mark(buf, it.line - 1, 0, { virt_lines = vl })
else
  mark(buf, it.end_line, 0, { virt_lines = vl })
end

A table stays put, and its lines stay lines of the buffer. All the plugin adds is spaces, as inline virtual text, so the pipes of every row line up under the pipes of the header. A column is as wide as its widest cell looks. A narrower cell gets the difference as spaces: after its text if the column is aligned left, before it if aligned right, half on each side if centred. The rule under the header is the one thing replaced, by dashes as wide as the column. The row under the cursor is left exactly as it was typed, like everything else the cursor is on.

Widths are measured with strdisplaywidth, the editor’s own answer to how many screen cells a string takes up. They’re kept with the item, so moving the cursor doesn’t measure every table again.

⟨line up the columns of a table⟩ ≡ used in the program
local t = it.table
local function looks(cell)
  return (cell.col - cell.from) + vim.fn.strdisplaywidth(cell.text) + (cell.to - cell.end_col)
end
if not it.widths then
  it.widths = {}
  for _, row in ipairs(t.rows) do
    for c, cell in ipairs(row.cells) do
      it.widths[c] = math.max(it.widths[c] or 3, looks(cell))
    end
  end
end
local function spaces(row, col, n)
  if n > 0 then
    mark(buf, row, col, { virt_text = { { string.rep(" ", n) } }, virt_text_pos = "inline" })
  end
end
for _, row in ipairs(t.rows) do
  if row.line ~= cursor then
    for _, h in ipairs(row.hide or {}) do
      mark(buf, row.line, h[1], { end_col = h[2], conceal = "" })
    end
    for c, cell in ipairs(row.cells) do
      local gap = (it.widths[c] or 0) - looks(cell)
      local before = ({ right = gap, center = math.floor(gap / 2) })[t.align[c]] or 0
      spaces(row.line, cell.from, before)
      spaces(row.line, cell.to, gap - before)
    end
  end
end
if it.line + 1 ~= cursor then
  for c, cell in ipairs(t.rule) do
    local w, a = it.widths[c] or 3, t.align[c]
    local dashes = (a == "center" and ":" or "-") .. string.rep("-", w - 2) .. (a == "left" and "-" or ":")
    mark(buf, it.line + 1, cell.from, { end_col = cell.to, conceal = "" })
    mark(buf, it.line + 1, cell.from, { virt_text = { { dashes, "LitgoTableRule" } }, virt_text_pos = "inline" })
  end
end
⟨draw inline math or a chunk name⟩ ≡ used in the program
local hl = ({ math = "LitgoMath", file = "LitgoFile", ["vego-math"] = "LitgoProof" })[it.kind] or "LitgoChunk"
if it.info == "undefined" then
  hl = "LitgoError"
end
if not under_cursor and it.text then
  mark(buf, it.line, it.col, { end_col = it.end_col, conceal = "" })
  mark(buf, it.line, it.col, { virt_text = { { it.text, hl } }, virt_text_pos = "inline" })
end
if it.info and it.info ~= "" then
  mark(buf, it.line, 0, { virt_text = { { "  " .. it.info, "LitgoInfo" } }, virt_text_pos = "eol" })
end

Analysis is asynchronous and debounced. An answer only counts if the buffer hasn’t changed since the question was asked. Stale news gets thrown away.

local function to_diagnostics(list, buf)
  local name = vim.api.nvim_buf_get_name(buf)
  local count = vim.api.nvim_buf_line_count(buf)
  local out = {}
  for _, d in ipairs(list or {}) do
    if not d.file or d.file == "" or d.file == name then
      out[#out + 1] = {
        lnum = math.min(d.line, count - 1),
        col = d.col or 0,
        end_col = (d.end_col and d.end_col > 0) and d.end_col or nil,
        severity = d.severity,
        message = d.message,
        source = d.source,
      }
    end
  end
  return out
end

M.to_diagnostics = to_diagnostics

--- Ask litgo about the buffer and redraw when the answer is still current.
function M.refresh(buf)
  local st = state[buf]
  local bin = litgo().bin()
  if not st or not bin or not vim.api.nvim_buf_is_valid(buf) then
    return
  end
  local tick = vim.api.nvim_buf_get_changedtick(buf)
  local cmd = { bin, "analyze", "--path", vim.api.nvim_buf_get_name(buf) }
  if litgo().config.render.upright then
    cmd[#cmd + 1] = "--upright"
  end
  local text = table.concat(vim.api.nvim_buf_get_lines(buf, 0, -1, false), "\n") .. "\n"
  vim.system(cmd, { stdin = text, text = true }, function(res)
    vim.schedule(function()
      if not vim.api.nvim_buf_is_valid(buf) then
        return
      end
      if vim.api.nvim_buf_get_changedtick(buf) ~= tick then
        -- The buffer moved on. Usually a debounced request is already queued;
        -- if not (the tick also bumps while a file finishes loading), ask again.
        if not st.dirty then
          M.refresh(buf)
        end
        return
      end
      local ok, decoded = pcall(vim.json.decode, res.stdout or "")
      if not ok or type(decoded) ~= "table" then
        return
      end
      st.items, st.chunks, st.dirty = decoded.items, decoded.chunks, false
      M.draw(buf)
      -- Reporting problems is the language server's job. Without it, the
      -- ones this analysis found are better than none; syntax errors while
      -- typing are noise, so they wait until insert mode ends.
      if require("litgo.lsp").active(buf) then
        vim.diagnostic.reset(check_ns, buf)
      elseif vim.api.nvim_get_mode().mode:sub(1, 1) ~= "i" then
        vim.diagnostic.set(check_ns, buf, to_diagnostics(decoded.diagnostics, buf))
      end
    end)
  end)
end

function M.schedule(buf)
  local st = state[buf]
  if not st then
    return
  end
  st.dirty = true
  if st.timer then
    st.timer:stop()
  else
    st.timer = vim.uv.new_timer()
  end
  st.timer:start(litgo().config.render.debounce_ms, 0, function()
    vim.schedule(function()
      M.refresh(buf)
    end)
  end)
end

function M.toggle(buf)
  local st = state[buf]
  if st then
    st.enabled = not st.enabled
    M.draw(buf)
  end
end

local function window_options(buf)
  for _, win in ipairs(vim.fn.win_findbuf(buf)) do
    vim.wo[win][0].conceallevel = 2
    vim.wo[win][0].concealcursor = ""
  end
end

function M.attach(buf)
  state[buf] = { enabled = litgo().config.render.enabled }
  local group = vim.api.nvim_create_augroup("litgo-render-" .. buf, { clear = true })
  vim.api.nvim_create_autocmd({ "TextChanged", "TextChangedI", "TextChangedP" }, {
    group = group,
    buffer = buf,
    callback = function()
      M.schedule(buf)
    end,
  })
  vim.api.nvim_create_autocmd("InsertLeave", {
    group = group,
    buffer = buf,
    callback = function()
      M.refresh(buf)
    end,
  })
  vim.api.nvim_create_autocmd({ "CursorMoved", "CursorMovedI", "BufEnter", "WinEnter" }, {
    group = group,
    buffer = buf,
    callback = function()
      local st = state[buf]
      -- While an analysis is pending the item positions are stale; the
      -- refresh that follows will redraw.
      if st and not st.dirty and st.cursor_row ~= vim.api.nvim_win_get_cursor(0)[1] - 1 then
        M.draw(buf)
      end
    end,
  })
  vim.api.nvim_create_autocmd("BufWinEnter", {
    group = group,
    buffer = buf,
    callback = function()
      window_options(buf)
    end,
  })
  vim.api.nvim_create_autocmd("BufWipeout", {
    group = group,
    buffer = buf,
    callback = function()
      local st = state[buf]
      if st and st.timer then
        st.timer:stop()
        st.timer:close()
      end
      state[buf] = nil
    end,
  })
  vim.diagnostic.config({ virtual_text = true, signs = true, underline = true, [require("litgo.proof").handler] = true }, check_ns)
  window_options(buf)
  M.refresh(buf)
end

return M

Tangle, weave, run

lua/litgo/run.lua
-- :Tangle, :Weave and :TangleCompileAndRun. The run loop streams JSON events
-- from `litgo run --json`; diagnostics arrive already translated to .lit.md
-- positions, so they can be placed in the buffer the programmer is editing.
local M = {}

local run_ns = vim.api.nvim_create_namespace("litgo-run")

M.ns = run_ns

local job -- the running vim.system object

local panel = { buf = nil }

local function litgo()
  return require("litgo")
end

Program output streams into a panel at the bottom. Its positions already point into the document: press <CR> on one to jump there, and q to close the panel.

While the program runs, the cursor stays where you were typing. A panel that steals the cursor mid-run is a panel that eats keystrokes. So the panel opens beside the work, hands the window back, and takes the cursor only once the run is over. The exception is a run that ended at an error, because then the cursor has somewhere better to be.

local function panel_buf()
  if panel.buf and vim.api.nvim_buf_is_valid(panel.buf) then
    return panel.buf
  end
  local buf = vim.api.nvim_create_buf(false, true)
  vim.api.nvim_buf_set_name(buf, "litgo://output")
  vim.bo[buf].buftype = "nofile"
  vim.bo[buf].bufhidden = "hide"
  vim.bo[buf].swapfile = false
  vim.bo[buf].filetype = "litgo-output"
  vim.keymap.set("n", "q", "<cmd>close<cr>", { buffer = buf, silent = true })
  -- <CR> on a `file.lit.md:12:3` reference jumps there.
  vim.keymap.set("n", "<cr>", function()
    local file, lnum, col = vim.api.nvim_get_current_line():match("([^%s:]+%.lit%.md):(%d+):?(%d*)")
    if not file then
      return
    end
    if vim.fn.filereadable(file) == 0 and panel.dir then
      file = panel.dir .. "/" .. file
    end
    local target = vim.fn.bufnr(file)
    for _, win in ipairs(vim.api.nvim_list_wins()) do
      if vim.api.nvim_win_get_buf(win) == target then
        vim.api.nvim_set_current_win(win)
        vim.api.nvim_win_set_cursor(win, { tonumber(lnum), math.max((tonumber(col) or 1) - 1, 0) })
        return
      end
    end
    vim.cmd("wincmd p | edit +" .. lnum .. " " .. vim.fn.fnameescape(file))
  end, { buffer = buf, silent = true })
  panel.buf = buf
  return buf
end

local function panel_win()
  local buf = panel_buf()
  for _, win in ipairs(vim.api.nvim_list_wins()) do
    if vim.api.nvim_win_get_buf(win) == buf then
      return win
    end
  end
  local current = vim.api.nvim_get_current_win()
  vim.cmd("botright " .. litgo().config.run.panel_height .. "split")
  local win = vim.api.nvim_get_current_win()
  vim.api.nvim_win_set_buf(win, buf)
  vim.wo[win].number = false
  vim.wo[win].relativenumber = false
  vim.wo[win].signcolumn = "no"
  vim.wo[win].winfixheight = true
  vim.wo[win].wrap = true
  vim.api.nvim_set_current_win(current)
  return win
end

--- Put the cursor in the panel, if it is on screen.
local function panel_focus()
  local buf = panel.buf
  if not (buf and vim.api.nvim_buf_is_valid(buf)) then
    return
  end
  for _, win in ipairs(vim.api.nvim_list_wins()) do
    if vim.api.nvim_win_get_buf(win) == buf then
      vim.api.nvim_set_current_win(win)
      return
    end
  end
end

local function panel_clear()
  local buf = panel_buf()
  vim.api.nvim_buf_set_lines(buf, 0, -1, false, {})
  vim.api.nvim_buf_clear_namespace(buf, run_ns, 0, -1)
end

local function panel_add(text, hl)
  local buf = panel_buf()
  if panel.dir then
    text = text:gsub(vim.pesc(panel.dir .. "/"), "") -- positions read better without the directory
  end
  local lines = vim.split(text, "\n", { plain = true })
  local count = vim.api.nvim_buf_line_count(buf)
  local empty = count == 1 and vim.api.nvim_buf_get_lines(buf, 0, 1, false)[1] == ""
  local first = empty and 0 or count
  vim.api.nvim_buf_set_lines(buf, first, -1, false, lines)
  if hl then
    for i = 0, #lines - 1 do
      vim.api.nvim_buf_set_extmark(buf, run_ns, first + i, 0, { line_hl_group = hl })
    end
  end
  local win = panel_win()
  vim.api.nvim_win_set_cursor(win, { vim.api.nvim_buf_line_count(buf), 0 })
end

Diagnostics from a run show up as virtual lines, right under the code that caused them, and they stay accurate while you edit. Change a line and its error goes away. Errors below an edit move with their text.

--- Place diagnostics in every file they mention and mirror them in quickfix.
local function publish(diags, origin)
  local by_file, qf = {}, {}
  for _, d in ipairs(diags) do
    local file = (d.file and d.file ~= "") and d.file or vim.api.nvim_buf_get_name(origin)
    by_file[file] = by_file[file] or {}
    table.insert(by_file[file], d)
    qf[#qf + 1] = {
      filename = file,
      lnum = d.line + 1,
      col = d.col + 1,
      text = d.message,
      type = d.severity == 1 and "E" or "W",
    }
  end
  for file, list in pairs(by_file) do
    local buf = vim.fn.bufadd(file)
    vim.fn.bufload(buf)
    -- What the compiler and the verifier say, the language server has usually
    -- said already.
    if require("litgo.lsp").reporting(buf) then
      list = vim.tbl_filter(function(d)
        return d.source ~= "go build" and d.source ~= "vego"
      end, list)
    end
    vim.diagnostic.set(run_ns, buf, require("litgo.render").to_diagnostics(list, buf))
  end
  vim.fn.setqflist({}, " ", { title = "litgo", items = qf })
end

function M.clear(buf)
  vim.diagnostic.reset(run_ns)
  if buf then
    vim.diagnostic.reset(run_ns, buf)
  end
end

--- Keep run diagnostics accurate while editing: fixing a line dismisses its
--- error, and errors below an edit move with their text.
function M.attach(buf)
  vim.diagnostic.config({
    virtual_lines = true, -- the message sits under the offending line
    virtual_text = false,
    signs = true,
    underline = true,
    severity_sort = true,
    [require("litgo.proof").handler] = true,
  }, run_ns)
  -- Several edits can arrive before the scheduled update runs (a macro, a
  -- :global command), so each works on the result of the one before.
  local pending
  vim.api.nvim_buf_attach(buf, false, {
    on_lines = function(_, _, _, first, last_old, last_new)
      local diags = pending or vim.diagnostic.get(buf, { namespace = run_ns })
      if #diags == 0 then
        return
      end
      local kept, changed = {}, false
      for _, d in ipairs(diags) do
        if d.lnum >= first and d.lnum < math.max(last_old, first + 1) then
          changed = true
        else
          if d.lnum >= last_old and last_new ~= last_old then
            d.lnum = d.lnum + last_new - last_old
            d.end_lnum = d.end_lnum and d.end_lnum + last_new - last_old
            changed = true
          end
          kept[#kept + 1] = d
        end
      end
      if changed and not pending then
        vim.schedule(function()
          if vim.api.nvim_buf_is_valid(buf) then
            vim.diagnostic.set(run_ns, buf, pending)
          end
          pending = nil
        end)
      end
      if changed then
        pending = kept
      end
    end,
  })
end
local function save(buf)
  if vim.bo[buf].modified then
    vim.api.nvim_buf_call(buf, function()
      vim.cmd("silent update")
    end)
  end
end

function M.tangle(buf, force)
  local bin = litgo().bin()
  if not bin then
    return
  end
  save(buf)
  local cmd = { bin, "tangle" }
  if force then
    cmd[#cmd + 1] = "--force"
  end
  cmd[#cmd + 1] = vim.api.nvim_buf_get_name(buf)
  vim.system(cmd, { text = true }, function(res)
    vim.schedule(function()
      local msg = vim.trim(res.stderr or "")
      vim.notify(msg ~= "" and msg or "litgo: tangled", res.code == 0 and vim.log.levels.INFO or vim.log.levels.ERROR)
    end)
  end)
end

function M.weave(buf, format)
  local bin = litgo().bin()
  if not bin then
    return
  end
  save(buf)
  local cmd = { bin, "weave" }
  if format and format ~= "" then
    vim.list_extend(cmd, { "--to", format })
  end
  cmd[#cmd + 1] = vim.api.nvim_buf_get_name(buf)
  vim.system(cmd, { text = true }, function(res)
    vim.schedule(function()
      if res.code ~= 0 then
        vim.notify(vim.trim(res.stderr or "litgo: weave failed"), vim.log.levels.ERROR)
        return
      end
      local out = vim.trim(res.stdout)
      vim.notify("litgo: wove " .. vim.fn.fnamemodify(out, ":~:."))
      if litgo().config.weave.open then
        vim.ui.open(out)
      end
    end)
  end)
end

--- Check the //@ annotations without compiling or running anything.
function M.prove(buf)
  local bin = litgo().bin()
  if not bin then
    return
  end
  save(buf)
  M.clear(buf)
  vim.system({ bin, "prove", "--json", vim.api.nvim_buf_get_name(buf) }, { text = true }, function(res)
    vim.schedule(function()
      local ok, decoded = pcall(vim.json.decode, res.stdout or "")
      if not ok or type(decoded) ~= "table" then
        vim.notify("litgo: prove failed\n" .. (res.stderr or ""), vim.log.levels.ERROR)
        return
      end
      publish(decoded.diagnostics or {}, buf)
      vim.g.litgo_last_proof = ("%d/%d"):format(decoded.proved, decoded.functions)
      local level = decoded.proved == decoded.functions and vim.log.levels.INFO or vim.log.levels.WARN
      if decoded.functions == 0 then
        vim.notify("litgo: no //@ annotations, so nothing to prove")
      else
        vim.notify(("litgo: ∴ proved %d of %d annotated functions (%d obligations)"):format(decoded.proved, decoded.functions, decoded.obligations), level)
      end
    end)
  end)
end

function M.stop()
  if job then
    job:kill("sigterm")
    job = nil
    panel_add("■ stopped", "LitgoError")
  end
end

function M.run(buf, args)
  local bin = litgo().bin()
  if not bin then
    return
  end
  if job then
    job:kill("sigterm")
    job = nil
  end
  save(buf)
  M.clear(buf)
  panel_clear()
  panel.dir = vim.fn.fnamemodify(vim.api.nvim_buf_get_name(buf), ":p:h")

  local cfg = litgo().config.run
  local cmd = { bin, "run", "--json" }
  if cfg.vet then
    cmd[#cmd + 1] = "--vet"
  end
  if cfg.timeout and cfg.timeout ~= "" then
    vim.list_extend(cmd, { "--timeout", cfg.timeout })
  end
  cmd[#cmd + 1] = vim.api.nvim_buf_get_name(buf)
  if args and #args > 0 then
    cmd[#cmd + 1] = "--"
    vim.list_extend(cmd, args)
  end

  local diags, pending = {}, ""
  local function handle(ev)
    if ev.event == "tangle" then
      panel_add("⟫ tangled → " .. ev.out, "LitgoInfo")
    elseif ev.event == "diagnostic" then
      diags[#diags + 1] = ev
      local where = ev.file and (ev.file .. ":" .. (ev.line + 1) .. ":" .. (ev.col + 1) .. ": ") or ""
      local hl = ev.severity == 1 and "LitgoError" or "DiagnosticWarn"
      if require("litgo.proof").is_proof(ev) then
        hl = ev.severity == 1 and "LitgoProofError" or "LitgoProofWarn"
      end
      panel_add(where .. ev.message, hl)
    elseif ev.event == "prove" then
      local all = ev.proved == ev.functions
      panel_add(("∴ proved %d of %d annotated functions (%d obligations, %d ms)"):format(ev.proved, ev.functions, ev.obligations, ev.ms), all and "LitgoProofKeyword" or "LitgoProofError")
    elseif ev.event == "build" then
      if ev.ok then
        panel_add(("✓ built in %d ms"):format(ev.ms), "LitgoOk")
      else
        local what = { tangle = "tangle failed", prove = "not proved, so not run" }
        panel_add("✗ " .. (what[ev.stage] or "build failed"), ev.stage == "prove" and "LitgoProofError" or "LitgoError")
      end
    elseif ev.event == "output" then
      panel_add(ev.text, ev.stream == "stderr" and "DiagnosticWarn" or nil)
    elseif ev.event == "exit" and ev.ran then
      local note = ev.timed_out and " (timed out)" or ""
      panel_add(("%s exit %d in %d ms%s"):format(ev.code == 0 and "✓" or "✗", ev.code, ev.ms, note), ev.code == 0 and "LitgoOk" or "LitgoError")
    end
  end

  local this
  this = vim.system(cmd, {
    text = true,
    stdout = function(_, data)
      if not data then
        return
      end
      vim.schedule(function()
        if job ~= this then
          return -- a newer run replaced this one
        end
        pending = pending .. data
        while true do
          local nl = pending:find("\n", 1, true)
          if not nl then
            break
          end
          local line = pending:sub(1, nl - 1)
          pending = pending:sub(nl + 1)
          local ok, ev = pcall(vim.json.decode, line)
          if ok and type(ev) == "table" then
            handle(ev)
          end
        end
      end)
    end,
  }, function()
    vim.schedule(function()
      if job ~= this then
        return
      end
      job = nil
      publish(diags, buf)
      local jumped = false
      if cfg.jump and vim.api.nvim_get_current_buf() == buf then
        local name = vim.api.nvim_buf_get_name(buf)
        for _, d in ipairs(diags) do
          if d.severity == 1 and (not d.file or d.file == name) then
            local row = math.min(d.line + 1, vim.api.nvim_buf_line_count(buf))
            vim.api.nvim_win_set_cursor(0, { row, d.col })
            vim.cmd("normal! zv")
            jumped = true
            break
          end
        end
      end
      if cfg.focus and not jumped and vim.api.nvim_get_current_buf() == buf then
        panel_focus()
      end
    end)
  end)
  job = this
end

return M

Errors as you type

lua/litgo/lsp.lua

Each .lit.md buffer gets the language server, and buffers under the same root share one. Neovim shows what the server publishes like any other diagnostics, except that the plugin asks for them in place, as virtual lines under the code, just like the diagnostics of a run.

The server brings gopls along, and all it needs from the plugin is help finding it. Most Neovim users get gopls from Mason, and Mason installs it somewhere only Neovim looks. After that, everything Neovim does for a buffer with a language server works here too: K, grr, grn, <C-x><C-o>, signature help on <C-s>, and whatever a completion plugin or a distribution adds.

Completion is the exception, because nothing switches it on by itself. nvim-cmp and blink.cmp find the server through the buffer it’s attached to, so they need nothing from the plugin. Plain Neovim only completes when asked, on <C-x><C-o>. So the plugin turns on Neovim’s own popup when no completion plugin is loaded, and stays out of the way when one is. Completion plugins usually load on the first insert, later than the server attaches, so the question gets asked again on the first insert, and answered only once.

local M = {}

--- The completion plugins that drive completion themselves.
local engines = { "blink.cmp", "cmp", "coq" }

local function engine_loaded()
  for _, name in ipairs(engines) do
    if package.loaded[name] then
      return true
    end
  end
  return false
end

--- Turn Neovim's own completion on for this buffer, unless something else
--- is doing the job. Deferred to the first insert, because that is when a
--- lazily loaded completion plugin arrives.
local function complete_with(id, buf)
  vim.api.nvim_create_autocmd("InsertEnter", {
    buffer = buf,
    once = true,
    callback = function()
      vim.schedule(function()
        if not engine_loaded() and vim.api.nvim_buf_is_valid(buf) then
          vim.lsp.completion.enable(true, id, buf, { autotrigger = true })
        end
      end)
    end,
  })
end

local function litgo()
  return require("litgo")
end

--- What to tell the server about gopls: where it is, to do without ("off"),
--- or to look for itself ("").
local function gopls(want)
  if want == false then
    return "off"
  elseif type(want) == "string" then
    return vim.fn.expand(want)
  end
  local mason = vim.fn.stdpath("data") .. "/mason/bin/gopls"
  if vim.fn.executable("gopls") == 0 and vim.fn.executable(mason) == 1 then
    return mason
  end
  return ""
end

function M.start(buf)
  local cfg = litgo().config.lsp
  local bin = cfg.enabled and litgo().bin()
  if not bin then
    return
  end
  local name = vim.api.nvim_buf_get_name(buf)
  vim.lsp.start({
    name = "litgo",
    cmd = { bin, "lsp", "--gopls", gopls(cfg.gopls) },
    init_options = cfg.settings,
    root_dir = vim.fs.root(name, { "go.mod", ".git" }) or vim.fs.dirname(name),
    on_attach = function(client, bufnr)
      vim.diagnostic.config({
        virtual_lines = cfg.virtual_lines,
        virtual_text = not cfg.virtual_lines,
        severity_sort = true,
        [require("litgo.proof").handler] = true,
      }, vim.lsp.diagnostic.get_namespace(client.id))
      if cfg.completion == "auto" then
        complete_with(client.id, bufnr)
      elseif cfg.completion then
        vim.lsp.completion.enable(true, client.id, bufnr, { autotrigger = true })
      end
    end,
  }, { bufnr = buf })
end

local function client(buf)
  return vim.lsp.get_clients({ bufnr = buf, name = "litgo" })[1]
end

--- Is the server attached to this buffer?
function M.active(buf)
  return client(buf) ~= nil
end

--- Is the server reporting errors in this buffer right now? If it is, a
--- run's compile errors would only repeat them.
function M.reporting(buf)
  local c = client(buf)
  return c ~= nil
    and #vim.diagnostic.get(buf, {
        namespace = vim.lsp.diagnostic.get_namespace(c.id),
        severity = vim.diagnostic.severity.ERROR,
      })
      > 0
end

return M

Proofs in their own colour

lua/litgo/proof.lua

Neovim colours a diagnostic by its severity and nothing else, so a failed proof would look like a compile error. It shouldn’t. It’s a different kind of news, and it calls for a different kind of fix: not code the compiler choked on, but an invariant that says too little.

Diagnostics are drawn by handlers, and handlers can be replaced. The ones that draw (signs, underlines, virtual text, virtual lines) get wrapped, so that in a .lit.md buffer they never see a diagnostic whose source is vego. One more handler draws exactly those: a violet undercurl, a ∴ in the sign column, and the message in violet under the line. In every other way they’re ordinary diagnostics, so ]d, the quickfix list and the location list all know about them.

-- Diagnostics from the verifier, drawn apart from the compiler's.
local M = {}

M.handler = "litgo/proof"

local drawn = {} -- diagnostic namespace -> extmark namespace

function M.is_proof(d)
  return type(d.source) == "string" and d.source:find("vego", 1, true) ~= nil
end

local function draw(buf, ns, d)
  local line = vim.api.nvim_buf_get_lines(buf, d.lnum, d.lnum + 1, false)[1] or ""
  local warn = d.severity ~= vim.diagnostic.severity.ERROR
  local hl = warn and "LitgoProofWarn" or "LitgoProofError"
  local pad = string.rep(" ", vim.fn.strdisplaywidth(line:match("^%s*")))
  local lines = {}
  for i, text in ipairs(vim.split(d.message, "\n")) do
    lines[i] = { { pad .. (i == 1 and "└─ ∴ " or "     ") .. text, hl } }
  end
  local end_col = math.min(d.end_col or #line, #line)
  pcall(vim.api.nvim_buf_set_extmark, buf, ns, d.lnum, math.min(d.col, #line), {
    end_col = math.max(end_col, math.min(d.col, #line)),
    hl_group = not warn and "LitgoProofUnderline" or nil,
    sign_text = "∴",
    sign_hl_group = hl,
    virt_lines = lines,
    priority = 220,
    strict = false,
  })
end

--- Install the handler, and keep the built-in ones away from proofs.
function M.setup()
  if vim.diagnostic.handlers[M.handler] then
    return
  end
  local function is_lit(buf)
    return require("litgo").is_lit(buf)
  end
  vim.diagnostic.handlers[M.handler] = {
    show = function(namespace, buf, diagnostics)
      drawn[namespace] = drawn[namespace] or vim.api.nvim_create_namespace("litgo-proof-" .. namespace)
      vim.api.nvim_buf_clear_namespace(buf, drawn[namespace], 0, -1)
      if not is_lit(buf) then
        return
      end
      for _, d in ipairs(diagnostics) do
        if M.is_proof(d) then
          draw(buf, drawn[namespace], d)
        end
      end
    end,
    hide = function(namespace, buf)
      if drawn[namespace] and vim.api.nvim_buf_is_valid(buf) then
        vim.api.nvim_buf_clear_namespace(buf, drawn[namespace], 0, -1)
      end
    end,
  }
  for _, name in ipairs({ "signs", "underline", "virtual_text", "virtual_lines" }) do
    local builtin = vim.diagnostic.handlers[name]
    if builtin then
      vim.diagnostic.handlers[name] = {
        show = function(namespace, buf, diagnostics, opts)
          if is_lit(buf) then
            diagnostics = vim.tbl_filter(function(d)
              return not M.is_proof(d)
            end, diagnostics)
          end
          builtin.show(namespace, buf, diagnostics, opts)
        end,
        hide = function(namespace, buf)
          if builtin.hide then
            builtin.hide(namespace, buf)
          end
        end,
      }
    end
  end
end

return M

Chunks

lua/litgo/chunks.lua
-- Chunk editing and navigation: :Untangle, :RenameChunk, gd and K.
local M = {}

local function litgo()
  return require("litgo")
end

local function refresh(buf)
  require("litgo.render").refresh(buf)
end

--- The chunk the cursor is on: a reference, a chunk directive, or the body of
--- a chunk block. Asks litgo afresh so it is right even mid-edit.
local function chunk_at_cursor(buf)
  local analysis, err = litgo().query(buf, { "analyze", "--path", vim.api.nvim_buf_get_name(buf) })
  if not analysis then
    return nil, err
  end
  local pos = vim.api.nvim_win_get_cursor(0)
  local row, col = pos[1] - 1, pos[2]
  local on_line
  for _, c in ipairs(analysis.chunks or {}) do
    for _, r in ipairs(c.refs) do
      if r.line == row then
        if col >= r.col and col < r.end_col then
          return c, "ref"
        end
        on_line = on_line or c
      end
    end
  end
  if on_line then
    return on_line, "ref"
  end
  for _, c in ipairs(analysis.chunks or {}) do
    for _, d in ipairs(c.defs) do
      if row == d.directive or (row >= d.first - 1 and row <= d.last + 1) then
        return c, "def"
      end
    end
  end
  return nil, "the cursor is not on a chunk or a chunk reference"
end

--- :[range]Untangle [name]
--- A charwise visual selection that stops short of whole lines becomes an
--- inline reference (a block comment); anything else a whole-line one.
function M.untangle(buf, o)
  local args = {
    "untangle",
    "--path", vim.api.nvim_buf_get_name(buf),
    "--start", tostring(o.line1 - 1),
    "--end", tostring(o.line2 - 1),
  }
  local s, e = vim.api.nvim_buf_get_mark(buf, "<"), vim.api.nvim_buf_get_mark(buf, ">")
  local from_visual = o.range == 2 and s[1] == o.line1 and e[1] == o.line2
  if from_visual and vim.fn.visualmode() == "v" then
    local last = vim.api.nvim_buf_get_lines(buf, e[1] - 1, e[1], false)[1] or ""
    local end_col = math.min(e[2], math.max(#last - 1, 0))
    -- '> is the first byte of the last selected character; step past it.
    end_col = end_col + #vim.fn.strcharpart(last:sub(end_col + 1), 0, 1)
    if vim.o.selection == "exclusive" then
      end_col = math.min(e[2], #last)
    end
    vim.list_extend(args, { "--start-col", tostring(s[2]), "--end-col", tostring(end_col) })
  end
  if o.args and o.args ~= "" then
    vim.list_extend(args, { "--name", o.args })
  end

  local res, err = litgo().query(buf, args)
  if not res then
    vim.notify("litgo: " .. err, vim.log.levels.ERROR)
    return
  end
  litgo().apply_edits(buf, res.edits)
  local target = litgo().config.untangle.jump and res.def_line or res.ref_line
  vim.api.nvim_win_set_cursor(0, { target + 1, 0 })
  vim.cmd("normal! ^")
  refresh(buf)
  vim.notify(("litgo: ⟨%s⟩ → line %d   (:RenameChunk to rename)"):format(res.name, res.def_line + 1))
end

--- :RenameChunk [new name] — renames every definition and every reference.
function M.rename(buf, new_name)
  local chunk, err = chunk_at_cursor(buf)
  if not chunk then
    vim.notify("litgo: " .. err, vim.log.levels.WARN)
    return
  end
  local function apply(name)
    if not name or vim.trim(name) == "" or name == chunk.name then
      return
    end
    local res, rename_err = litgo().query(buf, {
      "rename", "--path", vim.api.nvim_buf_get_name(buf), "--from", chunk.name, "--to", name,
    })
    if not res then
      vim.notify("litgo: " .. rename_err, vim.log.levels.ERROR)
      return
    end
    litgo().apply_edits(buf, res.edits)
    refresh(buf)
    vim.notify(("litgo: ⟨%s⟩ → ⟨%s⟩ (%d place%s)"):format(chunk.name, res.name, #res.edits, #res.edits == 1 and "" or "s"))
  end
  if new_name and new_name ~= "" then
    apply(new_name)
  else
    vim.ui.input({ prompt = "Rename ⟨" .. chunk.name .. "⟩ to: ", default = chunk.name }, apply)
  end
end

--- gd: from a reference to its definition; from a definition to its uses.
--- The language server does this better, because it does it for Go as well:
--- on a chunk it answers for itself, and anywhere else in a block, for gopls.
function M.goto_chunk()
  local buf = vim.api.nvim_get_current_buf()
  if require("litgo.lsp").active(buf) then
    vim.lsp.buf.definition()
    return
  end
  local chunk, where = chunk_at_cursor(buf)
  if not chunk then
    pcall(vim.cmd, "normal! gd")
    return
  end
  vim.cmd("normal! m'")
  if where == "ref" then
    if #chunk.defs == 0 then
      vim.notify("litgo: ⟨" .. chunk.name .. "⟩ is not defined", vim.log.levels.WARN)
      return
    end
    vim.api.nvim_win_set_cursor(0, { chunk.defs[1].first + 1, 0 })
  elseif #chunk.refs == 1 then
    vim.api.nvim_win_set_cursor(0, { chunk.refs[1].line + 1, chunk.refs[1].col })
  elseif #chunk.refs == 0 then
    vim.notify("litgo: ⟨" .. chunk.name .. "⟩ is never used", vim.log.levels.WARN)
  else
    local items = {}
    for _, r in ipairs(chunk.refs) do
      items[#items + 1] = {
        bufnr = buf,
        lnum = r.line + 1,
        col = r.col + 1,
        text = vim.trim(vim.api.nvim_buf_get_lines(buf, r.line, r.line + 1, false)[1] or ""),
      }
    end
    vim.fn.setloclist(0, {}, " ", { title = "uses of ⟨" .. chunk.name .. "⟩", items = items })
    vim.cmd("lopen")
  end
  vim.cmd("normal! zvzz")
end

--- K: preview the chunk behind a reference without leaving the host block.
--- This too is the language server's if it is there: a chunk on a reference,
--- gopls's documentation on a name.
function M.hover()
  local buf = vim.api.nvim_get_current_buf()
  if require("litgo.lsp").active(buf) then
    vim.lsp.buf.hover()
    return
  end
  local chunk, where = chunk_at_cursor(buf)
  if not chunk or where ~= "ref" or #chunk.defs == 0 then
    pcall(vim.cmd, "normal! K")
    return
  end
  local lines = { "⟨" .. chunk.name .. "⟩ ≡", "```go" }
  for _, d in ipairs(chunk.defs) do
    vim.list_extend(lines, vim.api.nvim_buf_get_lines(buf, d.first, d.last + 1, false))
  end
  lines[#lines + 1] = "```"
  vim.lsp.util.open_floating_preview(lines, "markdown", { border = "rounded", focus_id = "litgo-chunk" })
end

return M

Testing the plugin

Headless Neovim drives the real plugin against the real binary. The test covers rendering, both kinds of untangling, renaming, a run that succeeds, a run with an error inside a chunk, diagnostics that follow edits, a table whose columns get lined up, and a proof: annotations in their own colour, a broken invariant reported in violet while the compiler stays quiet, a run that refuses to go past it, and :Prove.

go build -o bin/litgo . && nvim --headless -u NONE -l test/plugin.lua
test/plugin.lua
-- Headless test of the plugin: nvim --headless -u NONE -l test/plugin.lua
local root = vim.fn.fnamemodify(debug.getinfo(1, "S").source:sub(2), ":p:h:h")
vim.opt.rtp:prepend(root)
local litgo = root .. "/bin/litgo"
assert(vim.fn.executable(litgo) == 1, "build bin/litgo first: go build -o bin/litgo .")
vim.o.swapfile = false

local failures = 0
local function check(cond, msg)
  if cond then
    print("ok   " .. msg)
  else
    failures = failures + 1
    print("FAIL " .. msg)
  end
end
local function wait(ms, cond, what)
  if not vim.wait(ms, cond, 20) then
    check(false, "timed out waiting for " .. what)
  end
end
local function find(buf, pattern)
  for i, l in ipairs(vim.api.nvim_buf_get_lines(buf, 0, -1, false)) do
    if l:find(pattern, 1, true) then
      return i
    end
  end
end

local dir = vim.fn.tempname()
vim.fn.mkdir(dir, "p")
local file = dir .. "/pool.lit.md"
vim.fn.writefile(vim.fn.readfile(root .. "/examples/worker_pool.lit.md"), file)

require("litgo").setup({ bin = litgo, weave = { open = false }, run = { jump = true } })
vim.cmd.edit(file)
local buf = vim.api.nvim_get_current_buf()
local render = require("litgo.render")
local run = require("litgo.run")

-- Rendering ------------------------------------------------------------------
wait(3000, function()
  local st = render.get(buf)
  return st and st.items ~= nil
end, "analysis")
local ns = vim.api.nvim_get_namespaces()["litgo-render"]
local marks = vim.api.nvim_buf_get_extmarks(buf, ns, 0, -1, { details = true })
local inline, blocks, concealed = 0, 0, 0
for _, m in ipairs(marks) do
  local d = m[4]
  if d.virt_text_pos == "inline" then inline = inline + 1 end
  if d.virt_lines then blocks = blocks + 1 end
  if d.conceal_lines then concealed = concealed + 1 end
end
check(inline >= 12, "inline math and chunk names are rendered (" .. inline .. ")")
check(blocks == 3, "two diagrams and one display formula are drawn (" .. blocks .. ")")
check(concealed == 3, "their source is concealed while the cursor is elsewhere")
check(#vim.diagnostic.get(buf) == 0, "a correct document has no diagnostics")

-- Moving onto a diagram reveals its source.
vim.api.nvim_win_set_cursor(0, { find(buf, "```mermaid") + 1, 0 })
vim.cmd("doautocmd CursorMoved")
concealed = 0
for _, m in ipairs(vim.api.nvim_buf_get_extmarks(buf, ns, 0, -1, { details = true })) do
  if m[4].conceal_lines then concealed = concealed + 1 end
end
check(concealed == 2, "the block under the cursor shows its source")

-- :Untangle, linewise ----------------------------------------------------------
local before = vim.fn.system({ litgo, "tangle", "--stdout", file })
local l1 = find(buf, "for job := range jobs")
vim.cmd(("%d,%dUntangle"):format(l1, l1 + 9))
check(vim.api.nvim_buf_get_lines(buf, l1 - 1, l1, false)[1] == "\t// <<for each job in jobs>>", "the selection became a reference")
check(find(buf, "<!-- chunk: for each job in jobs -->") ~= nil, "the chunk landed at the bottom")
vim.cmd("silent write")
check(vim.fn.system({ litgo, "tangle", "--stdout", file }) == before, "untangling leaves the program unchanged")

-- :Untangle, charwise (an expression) ------------------------------------------
local l2 = find(buf, "jobs := make(chan int, m)")
local line = vim.api.nvim_buf_get_lines(buf, l2 - 1, l2, false)[1]
local s = line:find("make", 1, true)
vim.api.nvim_win_set_cursor(0, { l2, s - 1 })
vim.cmd("normal! v$h")
vim.cmd("normal! \27")
vim.cmd("'<,'>Untangle")
check(vim.api.nvim_buf_get_lines(buf, l2 - 1, l2, false)[1] == "\tjobs := /*<<make call>>*/", "a charwise selection became an inline reference")
vim.cmd("silent write")
check(vim.fn.system({ litgo, "tangle", "--stdout", file }) == before, "inline untangling leaves the program unchanged")

-- :RenameChunk -------------------------------------------------------------------
vim.api.nvim_win_set_cursor(0, { l2, s + 5 })
vim.cmd("let &undolevels = &undolevels") -- scripts never pause for input, so close the undo block by hand
vim.cmd("RenameChunk the jobs channel")
check(find(buf, "/*<<the jobs channel>>*/") ~= nil and find(buf, "<!-- chunk: the jobs channel -->") ~= nil, "rename updates reference and definition")
check(find(buf, "make call") == nil, "no trace of the old name")
vim.cmd("normal u")
check(find(buf, "/*<<make call>>*/") ~= nil and find(buf, "<!-- chunk: make call -->") ~= nil, "rename is a single undo step")
vim.cmd("normal! \18") -- redo

-- gd ---------------------------------------------------------------------------
wait(30000, function() return require("litgo.lsp").active(buf) end, "the language server")
vim.api.nvim_win_set_cursor(0, { find(buf, "// <<feed the jobs>>"), 3 })
require("litgo.chunks").goto_chunk()
wait(10000, function() return vim.api.nvim_get_current_line() == "for i := 0; i < m; i++ {" end, "gd")
check(vim.api.nvim_get_current_line() == "for i := 0; i < m; i++ {", "gd jumps from a reference to the chunk")

-- Go in a Markdown buffer: gopls, behind the language server --------------------
local server = vim.lsp.get_clients({ bufnr = buf, name = "litgo" })[1]
if server and server.server_capabilities.completionProvider then
  local function ask(method, row, col)
    local params = { textDocument = vim.lsp.util.make_text_document_params(buf), position = { line = row - 1, character = col } }
    local answer = server:request_sync("textDocument/" .. method, params, 30000, buf)
    return answer and answer.result
  end
  -- wg is declared in main and used in a chunk: only the tangled program knows what it is.
  local l5 = find(buf, "wg.Add(1)")
  local hover
  wait(60000, function()
    hover = ask("hover", l5, 1)
    return hover ~= nil
  end, "gopls to load the program")
  check(hover and hover.contents.value:find("var wg sync.WaitGroup", 1, true) ~= nil, "hover in a chunk knows a variable of the block it is spliced into")
  check(hover and hover.range.start.line == l5 - 1 and hover.range.start.character == 1, "and answers about the Markdown")
  check(ask("hover", find(buf, "## The worker"), 4) == nil, "prose is not Go")

  vim.api.nvim_buf_set_lines(buf, l5, l5, false, { "\twg." })
  local items = ask("completion", l5 + 1, 4)
  local labels = {}
  for _, item in ipairs(items and (items.items or items) or {}) do
    labels[item.label] = item
  end
  check(labels.Wait ~= nil and labels.Done ~= nil, "completion in a chunk offers the methods of wg")
  local edit = labels.Wait and labels.Wait.textEdit or {}
  check((edit.insert or edit.range or { start = {} }).start.line == l5, "with edits that are edits of the Markdown")
  vim.api.nvim_buf_set_lines(buf, l5, l5 + 1, false, {})

  local l6 = find(buf, "go worker(ctx")
  vim.api.nvim_win_set_cursor(0, { l6, 4 })
  require("litgo.chunks").goto_chunk()
  wait(10000, function() return vim.api.nvim_get_current_line():find("^func worker%(") ~= nil end, "gd on a Go name")
  check(vim.api.nvim_get_current_line():find("^func worker%(") ~= nil, "gd on a Go name jumps to its definition, in another chunk")
else
  print("skip gopls is not installed")
end

-- :TangleCompileAndRun, success ------------------------------------------------
vim.cmd("silent write")
vim.cmd("TangleCompileAndRun")
local function panel_text()
  local pb = vim.fn.bufnr("litgo://output")
  return pb > 0 and table.concat(vim.api.nvim_buf_get_lines(pb, 0, -1, false), "\n") or ""
end
wait(60000, function() return panel_text():find("exit %d+ in") ~= nil end, "the program to finish")
check(panel_text():find("jobs finished") ~= nil, "program output reaches the panel")
check(panel_text():find("✓ built") ~= nil, "the build is reported")
check(#vim.diagnostic.get(buf, { namespace = run.ns }) == 0, "a clean run leaves no diagnostics")

-- The panel has the cursor when the run is over, and q gives it back.
local function panel_win()
  local pb = vim.fn.bufnr("litgo://output")
  for _, w in ipairs(vim.api.nvim_list_wins()) do
    if vim.api.nvim_win_get_buf(w) == pb then
      return w
    end
  end
end
wait(10000, function() return vim.api.nvim_get_current_win() == panel_win() end, "the panel to take the cursor")
check(vim.api.nvim_get_current_win() == panel_win(), "the output panel takes the cursor when the run ends")
vim.api.nvim_feedkeys("q", "x", false)
check(panel_win() == nil, "q closes it")
check(vim.api.nvim_get_current_buf() == buf, "and the cursor is back in the document")

-- Errors as you type: the language server, nothing run -------------------------
local function diag(pattern, ns)
  for _, x in ipairs(vim.diagnostic.get(buf, { namespace = ns })) do
    if x.message:find(pattern, 1, true) then
      return x
    end
  end
end
check(require("litgo.lsp").active(buf), "the language server is attached")

-- What a completion plugin needs to keep Markdown snippets out of Go blocks.
local blocks = require("litgo")
check(blocks.in_go_block(buf, find(buf, "fmt.Println(r)") - 1), "a line of Go is in a Go block")
check(not blocks.in_go_block(buf, find(buf, "# Worker pool") - 1), "a heading is not")
check(not blocks.in_go_block(buf, find(buf, "flowchart LR") - 1), "and neither is a diagram")

local l3 = find(buf, "fmt.Println(r)")
vim.api.nvim_buf_set_lines(buf, l3 - 1, l3, false, { "\tfmt.Println(rr)" })
wait(60000, function() return diag("undefined: rr") ~= nil end, "the language server to find the type error")
local d = diag("undefined: rr")
check(d and d.lnum == l3 - 1 and d.col == 13 and d.end_col == 15, "a type error inside a chunk is underlined where it is, unsaved and unrun")
local first = diag("declared and not used: r")
check(first ~= nil, "and so is the variable that is now unused")

-- A run adds nothing the server has said, but still jumps and fills quickfix.
vim.api.nvim_win_set_cursor(0, { 1, 0 })
vim.cmd("TangleCompileAndRun")
wait(60000, function() return panel_text():find("build failed") ~= nil end, "the build to fail")
wait(2000, function() return #vim.fn.getqflist() == 2 end, "the quickfix list")
check(vim.api.nvim_get_current_win() ~= panel_win(), "a run that fails leaves the cursor in the document")
check(first and vim.api.nvim_win_get_cursor(0)[1] == first.lnum + 1, "the cursor jumped to the first error")
check(#vim.fn.getqflist() == 2, "the quickfix list mirrors the compiler")
check(diag("undefined: rr", run.ns) == nil, "the run does not repeat what the server reports")

vim.api.nvim_buf_set_lines(buf, l3 - 1, l3, false, { "\tfmt.Println(r)" })
wait(20000, function() return diag("undefined: rr") == nil and diag("declared and not used") == nil end, "the errors to clear")
check(#vim.diagnostic.get(buf) == 0, "fixing the line clears the errors")

-- Diagnostics from a run (a panic, a failing test) stay accurate between runs.
vim.diagnostic.set(run.ns, buf, {
  { lnum = l3 - 1, col = 0, message = "panic: one", severity = 1 },
  { lnum = l3 + 1, col = 0, message = "panic: two", severity = 1 },
})
vim.api.nvim_buf_set_lines(buf, l3 - 1, l3, false, { "\tfmt.Println(r) " })
vim.api.nvim_buf_set_lines(buf, 0, 0, false, { "", "" })
wait(2000, function() local x = diag("panic: two", run.ns) return x and x.lnum == l3 + 3 end, "run diagnostics to move")
check(diag("panic: one", run.ns) == nil, "editing a line dismisses the run diagnostic on it")
check(diag("panic: two", run.ns).lnum == l3 + 3, "and the ones below follow their lines")
vim.api.nvim_buf_set_lines(buf, 0, 2, false, {})
run.clear(buf)

-- Structure is the server's business too.
local l4 = find(buf, "// <<spawn the workers>>")
vim.api.nvim_buf_set_lines(buf, l4 - 1, l4, false, { "\t// <<spawn teh workers>>" })
wait(20000, function() return diag("is not defined") ~= nil and diag("is never used") ~= nil end, "the structural diagnostics")
check(diag("spawn teh workers⟩ is not defined") and diag("spawn the workers⟩ is never used"), "a misspelt reference is flagged at both ends")
vim.cmd("normal u")

-- :Weave ------------------------------------------------------------------------
if vim.fn.executable("pandoc") == 1 then
  vim.cmd("silent write")
  vim.cmd("Weave html")
  wait(20000, function() return vim.uv.fs_stat(dir .. "/pool.html") ~= nil end, "the woven page")
  check(vim.uv.fs_stat(dir .. "/pool.html") ~= nil, ":Weave html writes pool.html")
  if vim.fn.executable("typst") == 1 and vim.system({ "typst", "--version" }):wait().code == 0 then
    vim.cmd("Weave")
    wait(30000, function() return vim.uv.fs_stat(dir .. "/pool.pdf") ~= nil end, "the woven PDF")
    check(vim.uv.fs_stat(dir .. "/pool.pdf") ~= nil, ":Weave writes pool.pdf")
  end
end

-- Tables ---------------------------------------------------------------------------
vim.cmd("enew!")
local tbuf = vim.api.nvim_get_current_buf()
vim.api.nvim_buf_set_name(tbuf, dir .. "/table.lit.md")
vim.api.nvim_buf_set_lines(tbuf, 0, -1, false, {
  "# Keys",
  "",
  "| Key | What it does | Cost |",
  "|:-:|---|--:|",
  "| `K` | shows the *documentation* | $x^2$ |",
  "| `<C-x><C-o>` | completes | 10 |",
  "",
  "The end.",
})
require("litgo").attach(tbuf)
wait(3000, function()
  local st = render.get(tbuf)
  return st and st.items ~= nil
end, "the analysis of the table")

-- What a row looks like: the text that is not concealed, and the virtual
-- text where it is put in.
local function shown(row)
  local text = vim.api.nvim_buf_get_lines(tbuf, row, row + 1, false)[1]
  local hidden, put = {}, {}
  for _, m in ipairs(vim.api.nvim_buf_get_extmarks(tbuf, ns, { row, 0 }, { row, -1 }, { details = true })) do
    local d = m[4]
    if d.conceal then
      for i = m[3] + 1, d.end_col do hidden[i] = true end
    elseif d.virt_text_pos == "inline" then
      put[m[3]] = (put[m[3]] or "") .. d.virt_text[1][1]
    end
  end
  local out = ""
  for i = 1, #text + 1 do
    out = out .. (put[i - 1] or "")
    if i <= #text and not hidden[i] then out = out .. text:sub(i, i) end
  end
  return out
end
check(vim.api.nvim_buf_line_count(tbuf) == 8 and #vim.api.nvim_buf_get_extmarks(tbuf, ns, 0, -1, { details = true }) > 0,
  "a table stays where it is")
check(shown(2) == "|    Key     | What it does            | Cost |", "the header is padded, and its first column is centred: " .. shown(2))
check(shown(3) == "|:----------:|-------------------------|-----:|", "the rule is as wide as the columns: " .. shown(3))
check(shown(4) == "|     K      | shows the documentation |   𝑥² |", "cells are as wide as they look, not as their source: " .. shown(4))
check(shown(5) == "| <C-x><C-o> | completes               |   10 |", "the widest cell decides: " .. shown(5))

-- The same thing as the screen shows it, which is what counts.
vim.cmd("redraw")
local screen = {}
for r = 3, 6 do
  local s = ""
  for c = 1, 60 do s = s .. vim.fn.screenstring(r, c) end
  screen[#screen + 1] = vim.trim(s)
end
check(screen[3] == shown(4) and screen[4] == shown(5), "and that is what is on the screen: " .. screen[3])
vim.api.nvim_win_set_cursor(0, { 5, 0 })
vim.cmd("doautocmd CursorMoved")
check(shown(4) == "| `K` | shows the *documentation* | $x^2$ |", "the row under the cursor is left as it was typed")
check(shown(5):find("completes               |", 1, true) ~= nil, "and the other rows stay lined up")
vim.cmd("bwipeout!")

vim.fn.delete(dir, "rf")
-- Proofs ---------------------------------------------------------------------------
local pdir = vim.fn.tempname()
vim.fn.mkdir(pdir, "p")
local pfile = pdir .. "/isqrt.lit.md"
vim.fn.writefile(vim.fn.readfile(root .. "/examples/isqrt.lit.md"), pfile)
vim.cmd.edit(pfile)
local pbuf = vim.api.nvim_get_current_buf()
wait(3000, function()
  local st = render.get(pbuf)
  return st and st.items ~= nil
end, "analysis of the proved example")
local keywords, formulas = 0, 0
for _, m in ipairs(vim.api.nvim_buf_get_extmarks(pbuf, ns, 0, -1, { details = true })) do
  if m[4].hl_group == "LitgoProofKeyword" then keywords = keywords + 1 end
  if m[4].hl_group == "LitgoProof" then formulas = formulas + 1 end
end
check(keywords == 5 and formulas == 5, "the five annotations have a colour of their own (" .. keywords .. ")")
local inv = find(pbuf, "//@ Invariant lo*lo")
local km = vim.api.nvim_buf_get_extmarks(pbuf, ns, { inv - 1, 0 }, { inv - 1, -1 }, { details = true })[1]
check(km and km[3] == 0 and km[4].end_col == #"//@ Invariant", "the keyword is set apart from the formula")

-- The formula is typeset like the math in the prose, except under the cursor.
local function typeset(row)
  local text, hidden
  for _, m in ipairs(vim.api.nvim_buf_get_extmarks(pbuf, ns, { row, 0 }, { row, -1 }, { details = true })) do
    if m[4].virt_text_pos == "inline" then text = m[4].virt_text[1][1] end
    if m[4].conceal then hidden = { m[3], m[4].end_col } end
  end
  return text, hidden
end
vim.api.nvim_win_set_cursor(0, { 1, 0 })
render.draw(pbuf)
local text, hidden = typeset(inv - 1)
check(text == "𝑙𝑜² ≤ 𝑛 ∧ 𝑛 < ℎ𝑖²", "a formula is typeset: " .. tostring(text))
check(hidden and hidden[1] == #"//@ Invariant " and hidden[2] == #"//@ Invariant lo*lo <= n ^ n < hi*hi", "in place of its source, and the keyword stays")
vim.api.nvim_win_set_cursor(0, { inv, 0 })
render.draw(pbuf)
check(typeset(inv - 1) == nil, "and the source comes back under the cursor")
vim.api.nvim_win_set_cursor(0, { 1, 0 })

local function proof_marks()
  local out = {}
  for name, id in pairs(vim.api.nvim_get_namespaces()) do
    if name:find("litgo-proof-", 1, true) then
      vim.list_extend(out, vim.api.nvim_buf_get_extmarks(pbuf, id, 0, -1, { details = true }))
    end
  end
  return out
end
local function proofs()
  return vim.tbl_filter(require("litgo.proof").is_proof, vim.diagnostic.get(pbuf))
end
wait(10000, function() return require("litgo.lsp").active(pbuf) end, "the server on the proved example")
check(#proofs() == 0 and #proof_marks() == 0, "a proved document has nothing violet under it")

-- Break the bisection. It still compiles, so only the verifier can object.
local lo = find(pbuf, "\tlo = mid")
vim.api.nvim_buf_set_lines(pbuf, lo - 1, lo, false, { "\tlo = mid + 1" })
wait(15000, function() return #proofs() == 2 end, "the verifier's objections")
local failed = proofs()
check(#failed == 2 and failed[1].lnum == inv - 2 and failed[1].message:find("Invariant is not maintained", 1, true) ~= nil,
  "a broken invariant is reported at the invariant, as you type")
check(#vim.diagnostic.get(pbuf) == 2, "and the compiler has nothing to add")
local violet = proof_marks()
check(#violet == 2 and violet[1][4].hl_group == "LitgoProofUnderline" and violet[1][4].virt_lines[1][1][2] == "LitgoProofError"
  and vim.trim(violet[1][4].sign_text) == "∴", "it is drawn in the proof colour, not the compiler's")
-- Neovim keeps marks of its own to follow a diagnostic through edits; what
-- counts is a mark that draws something.
local builtin = 0
for name, id in pairs(vim.api.nvim_get_namespaces()) do
  if not name:find("litgo-", 1, true) then
    for _, m in ipairs(vim.api.nvim_buf_get_extmarks(pbuf, id, 0, -1, { details = true })) do
      local d = m[4]
      if d.hl_group or d.virt_text or d.virt_lines or d.sign_text then
        builtin = builtin + 1
      end
    end
  end
end
check(builtin == 0, "and the built-in handlers leave it alone")

-- A run refuses to go further than the proof.
vim.cmd("TangleCompileAndRun")
wait(20000, function() return panel_text():find("not proved, so not run", 1, true) ~= nil end, "the run that is not proved")
check(panel_text():find("proved 0 of 1", 1, true) ~= nil, "a run stops at a failed proof")
check(panel_text():find("isqrt(", 1, true) == nil, "and the program was not run")

vim.api.nvim_set_current_buf(pbuf)
vim.api.nvim_buf_set_lines(pbuf, lo - 1, lo, false, { "\tlo = mid" })
wait(15000, function() return #proofs() == 0 and #proof_marks() == 0 end, "the proof to come back")
check(#proofs() == 0, "fixing the code takes the violet away")
vim.cmd("Prove")
wait(10000, function() return vim.g.litgo_last_proof ~= nil end, ":Prove")
check(vim.g.litgo_last_proof == "1/1", ":Prove proves it without running it")

-- A new file starts as a program that runs ------------------------------------
vim.cmd.edit(dir .. "/hello.lit.md")
local nbuf = vim.api.nvim_get_current_buf()
check(find(nbuf, "# hello") ~= nil, "a new file is titled after its name")
check(find(nbuf, "<!-- imports: fmt -->") ~= nil, "and has the directives")
check(find(nbuf, "func main() {") ~= nil, "and a main")
vim.wait(3000, function() return false end, 20)
check(#vim.diagnostic.get(nbuf) == 0, "and no diagnostics")
local kept = vim.fn.tempname() .. ".lit.md"
vim.fn.writefile({ "# kept" }, kept)
vim.cmd.edit(kept)
check(#vim.api.nvim_buf_get_lines(0, 0, -1, false) == 1, "a file with something in it is left alone")

print(failures == 0 and "\nall passed" or ("\n" .. failures .. " failed"))
os.exit(failures == 0 and 0 or 1)

An example

Here’s a complete literate program, small enough to read over coffee. The tests above use it, and it’s the best first file to open in the editor. It’s a document inside a document, so its fence is one backtick longer than the fences it contains. It’s also verbatim, because its references belong to it, not to this document.

examples/worker_pool.lit.md
<!-- tangler: go -->
<!-- package: main -->
<!-- imports: context, fmt, sync, time -->
<!-- tangler-exclude: ^\s*// NOTE.*$ -->

# Worker pool in Go

Three workers, ten jobs, one deadline. A pool of $n$ workers chews through $m$
jobs at the same time. The jobs travel through a channel, and the context can
be cancelled, so we can stop early and still keep whatever already finished.

If every job costs $t$ seconds, the pool finishes in about

$$
T(n, m) = \left\lceil \frac{m}{n} \right\rceil \cdot t
$$

instead of the $m \cdot t$ one goroutine would need. That's a speed-up of
$\frac{m t}{T} \approx n$, for as long as $n \le m$.

```mermaid
flowchart LR
    main -->|jobs| w1[worker 1] & w2[worker 2] & w3[worker 3]
    w1 & w2 & w3 -->|results| collect
```

## The shape of the program

The whole program fits on one screen, because every step is a named chunk
explained further down. Chunks are spliced in *textually*, so they see `ctx`,
`jobs`, `results` and `wg` exactly as if they'd been written right here.

```go
// <<the worker>>

func main() {
    // <<set up a cancellable context>>

    n, m := 3, 10
    jobs := make(chan int, m)
    results := make(chan int, m)
    var wg sync.WaitGroup

    // <<spawn the workers>>
    // <<feed the jobs>>
    // <<close results once the workers are done>>
    // <<collect the results>>
}
```

## The worker

A worker drains `jobs` until the channel closes or the context is cancelled,
whichever comes first. Doubling a number stands in for real work. This worker
is overpaid.

<!-- chunk: the worker -->
```go
func worker(ctx context.Context, id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
    defer wg.Done()
    for job := range jobs {
        select {
        case <-ctx.Done():
            return
        default:
            // NOTE: this line is dropped by the tangler-exclude directive.
            time.Sleep(20 * time.Millisecond)
            results <- job * 2
        }
    }
}
```

```mermaid
sequenceDiagram
    participant M as main
    participant W as worker
    M->>W: job
    W-->>M: job * 2
    M-xW: cancel
```

## Cancellation

The deadline is a tiny chunk of its own, an expression spliced into the middle
of a line.

<!-- chunk: set up a cancellable context -->
```go
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
time.AfterFunc(/*<<the deadline>>*/, cancel)
```

Fifty milliseconds. Long enough for roughly half the jobs, so you get to watch
the cancellation actually cut something off.

<!-- chunk: the deadline -->
```go
50 * time.Millisecond
```

## Spawn, feed, collect

<!-- chunk: spawn the workers -->
```go
for i := 0; i < n; i++ {
    wg.Add(1)
    go worker(ctx, i, jobs, results, &wg)
}
```

The channel has room for all $m$ jobs, so feeding it never blocks.

<!-- chunk: feed the jobs -->
```go
for i := 0; i < m; i++ {
    jobs <- i
}
close(jobs)
```

<!-- chunk: close results once the workers are done -->
```go
go func() {
    wg.Wait()
    close(results)
}()
```

Ranging over `results` ends when the channel closes, and that happens after the
last worker returns, whether it ran out of jobs or got cancelled.

<!-- chunk: collect the results -->
```go
done := 0
for r := range results {
    fmt.Println(r)
    done++
}
fmt.Printf("%d of %d jobs finished\n", done, m)
```

An example with a proof

The second example is about as small as a proved program can be and still say something: an integer square root by bisection, with a contract, an invariant and a variant. Open it to see what Proofs looks like from the outside. The tests prove it, and the plugin’s test breaks it on purpose to watch the editor object.

examples/isqrt.lit.md
<!-- tangler: go -->
<!-- package: main -->
<!-- imports: fmt -->

# A square root you can trust

The integer square root of $n$ is the largest integer whose square doesn't go
over $n$:

$$
r = \lfloor \sqrt{n} \rfloor \quad\Longleftrightarrow\quad r^2 \le n < (r+1)^2
$$

This program finds it by bisection, and it comes with a proof. Not a test that
it gets $\sqrt{17}$ right. A proof. A proof that the inequality on the right
holds for the result of *every* call with $n \ge 0$, that the loop always
stops, and that the division in it can never divide by zero. The proof lives in
the `//@` comments, written in the notation of
[VeGo](https://arxiv.org/abs/2608.22630), and `litgo run` checks it before it
compiles a single line:

```sh
litgo run   examples/isqrt.lit.md    # prove, then compile, then run
litgo prove examples/isqrt.lit.md    # only prove
```

```mermaid
flowchart LR
    doc[isqrt.lit.md] -->|tangle| go[isqrt.go]
    go -->|prove| ok{proved?}
    ok -->|yes| build[go build, run]
    ok -->|no| line[a violet line under the annotation that failed]
```

## The contract

The function makes one promise and asks for one thing in return. `Requires` is
what the caller owes, `Ensures` is what the function owes back, and the result
gets a name, `r`, so the promise can talk about it.

```go
//@ Requires n >= 0
func Isqrt(n int) (r int) {
    // <<bisect>>
}
//@ Ensures r*r <= n ^ n < (r+1)*(r+1)
```

`^` means *and*. The `Ensures` sits under the closing brace, which is where a
conclusion belongs.

## The loop

Bisection keeps two numbers: one whose square is known to be small enough, one
whose square is known to be too big. It moves one of them to the middle, over
and over, until they're neighbours.

<!-- chunk: bisect -->
```go
lo, hi := 0, n+1
//@ Invariant 0 <= lo < hi
//@ Invariant lo*lo <= n ^ n < hi*hi
//@ Variant hi - lo
for hi-lo > 1 {
    mid := lo + (hi-lo)/2
    // <<move whichever end the middle can replace>>
}
return lo
```

The sentence above the loop *is* the invariant: `lo*lo <= n ^ n < hi*hi`. A
loop invariant is something true before the loop and true again after every
trip around it, and those are exactly the two things the verifier checks. It's
true before, because $0 \le n$ and $n < (n+1)^2$. Then the verifier forgets
everything it knew about `lo` and `hi` except the invariant, goes around once,
and has to win the invariant back.

<!-- chunk: move whichever end the middle can replace -->
```go
if mid*mid <= n {
    lo = mid
} else {
    hi = mid
}
```

Whichever branch runs, the test it just made is word for word the half of the
invariant it has to restore. That part comes free. The other line of the
invariant, `0 <= lo < hi`, needs the middle to land strictly between the ends,
and that's where the loop condition earns its keep: `hi - lo > 1` means
`(hi-lo)/2` is at least 1 and less than `hi - lo`.

The same fact proves the loop stops. The `Variant` is a quantity that never
goes negative while the loop runs and shrinks on every round. Here it's the
width of the interval, and both branches squeeze it.

After the loop, the verifier knows two things: the invariant, and that the
condition is false. From `lo < hi` and not `hi - lo > 1`, it follows that
`hi = lo + 1`. Plug that into `n < hi*hi` and out comes the `Ensures`. Nobody
had to spell that out. A few lines of arithmetic like that are exactly what the
prover is for.

## What the proof does not say

Okay, the fine print. The integers of the proof are the integers of
mathematics, and they go on forever. Go's `int` has 64 bits, and `(n+1)*(n+1)`
overflows long before `n` runs out of them. For $n < 3 \cdot 10^9$ the two
agree. Beyond that, the theorem is about a program this one only resembles.
VeGo makes the same simplification, and it's the kind of thing you want to
know about a proof before you lean on it.

## Running it

`main` has no annotations, so the verifier leaves it alone. You can prove a
program one function at a time.

```go
func main() {
    for _, n := range []int{0, 1, 2, 15, 16, 17, 1_000_000, 2_147_395_599} {
        fmt.Printf("isqrt(%d) = %d\n", n, Isqrt(n))
    }
}
```

## Breaking it

The quickest way to believe a verifier is to lie to it. Change `lo = mid` to
`lo = mid + 1`, which looks like textbook bisection and is wrong here, then run
it again:

```text
isqrt.lit.md:61:1: error: Invariant is not maintained by the body of the loop at line 64: 0 <= lo < hi
isqrt.lit.md:62:1: error: Invariant is not maintained by the body of the loop at line 64: lo*lo <= n ^ n < hi*hi
litgo: proved 0 of 1 annotated functions in isqrt.lit.md (11 obligations, 79 ms)
litgo: not proved, so not run (--no-prove runs it anyway)
```

Both complaints are right. The test only established `mid*mid <= n`, not
`(mid+1)*(mid+1) <= n`, and `mid + 1` can be `hi` itself, which leaves the
interval empty. In the editor these two lines turn violet, not red, because the
program still compiles. The syntax is fine. It's the argument that's broken.

Or loosen the loop condition to `hi-lo > 0`, and the verifier points at the
`Variant`: when the interval is one wide the middle is `lo`, nothing moves, and
the loop spins forever. A test would have hung. The proof just says no.

A tutorial in proofs

The third example is a tour of every annotation the verifier checks, one small function at a time, and it ends with binary search. The tests prove every function in it, so the tutorial can’t drift away from the verifier it describes.

examples/vego-tutorial.lit.md
<!-- tangler: go -->
<!-- package: main -->
<!-- imports: fmt -->

# VeGo in litgo: a tutorial

[VeGo](https://arxiv.org/abs/2608.22630) (Verified Go, by Tina Massoudi and
Chris Dutchyn) lets you write down what a Go function promises, and why its
loops are right, in comments that start with `//@`. The program stays ordinary
Go. litgo reads the comments and proves them, or tells you exactly which one it
couldn't prove, and for which line of code.

This is a tour of every annotation litgo checks, one small function at a time.
Every function here is proved each time the document runs:

```sh
litgo prove examples/vego-tutorial.lit.md   # prove, and list what was proved
litgo run   examples/vego-tutorial.lit.md   # prove, then compile and run
litgo check examples/vego-tutorial.lit.md   # chunks, syntax, types, proofs
```

In Neovim the annotations are violet and their formulas are typeset: `^`
reads ∧, `<=` reads ≤ and `Forall` reads ∀, until the cursor lands on the line
and you get back what you typed. Whatever the verifier has to say about them
shows up in violet too, as you type. A red line means the compiler won't take
the program. A violet line means the program runs, but nobody has shown that it
keeps its word.

The best way to read this is to break things. Change a `<` to a `<=` in any
block and watch what the verifier says.

```mermaid
flowchart LR
    req[Requires] --> body[the body, one path at a time]
    inv[Invariant, Variant] --> body
    body --> ens[Ensures at every return]
    body --> safe[every index in range, every divisor not zero]
    body --> calls[the Requires of every call]
```

## A contract

`Requires` is what the caller owes. `Ensures` is what the function owes back.
`Requires` goes above the function, and `Ensures` goes directly under its
closing brace, where a conclusion belongs. The result needs a name, so the
`Ensures` has something to call it.

```go
//@ Requires x > 0 ^ y > 0
func AddPositives(x, y int) (sum int) {
    return x + y
}
//@ Ensures sum > x ^ sum > y
```

Without the `Requires` this is false (try `0` and `0`), and the verifier
catches it: *Ensures is not proved for the return at line 50*.

A function with no annotations is left alone. That means you can prove a
program one function at a time, and `main` below never gets looked at.

One warning before your editor surprises you. gofmt rewrites `//@` in a doc
comment as `// @`, and puts a blank line above an `Ensures`. litgo reads both
spellings and looks upward past the blank line, so formatting on save changes
how a contract looks, not what it means.

## The language of formulas

| | |
|:--|:--|
| `^`  `v`  `~` | and, or, not |
| `->`  `<-`  `<->` | implies, is implied by, if and only if |
| `=`  `<>`  `<`  `<=`  `>`  `>=` | comparisons. They chain: `0 <= i < n`. Go's `==` and `!=` are accepted too. |
| `+`  `-`  `*`  `/`  `%` | integer arithmetic; `/` and `%` are Go's, which round towards zero |
| `len(A)`, `A[i]` | the length and the elements of a slice of integers |
| `s.Len` | an integer or boolean field of a struct |
| `Forall k in [a, b) : P` | for every integer `k` with `a <= k < b` |
| `Exists k in [a, b] : P` | for some `k` with `a <= k <= b` |
| `Unique k in (a, b) : P` | for exactly one `k` with `a < k < b` |
| `Z`, `[0, ...)` | all integers; an interval without an upper end |
| `A[a:b) <= m` | every element of that range of `A` is at most `m` |
| `m in A[a:b)`, `m = A[a:b)` | some element of the range is `m` |
| `x'` | `x` after an assignment, where `x` is before it |

Mathematicians write the length of a slice as `|A|`, and so does the rest of
this tutorial. Go's own `&&`, `||` and `!` work too, for *and*, *or* and *not*.

A formula too long for one line continues in the next `//@` comment, as long as
its line ends in something that can't end a formula, like an operator.

## Safety comes for free

Some obligations nobody writes down, because the code implies them. Every
index has to land inside its slice. Every divisor has to be nonzero. This
function needs its `Requires` for no other reason:

```go
//@ Requires 0 <= i ^ i < |A| ^ d <> 0
func ElementOver(A []int, i, d int) (q int) {
    return A[i] / d
}
//@ Ensures q * d + A[i] % d = A[i]
```

Take away `i < |A|` and the complaint lands on the `A[i]` itself: *the index
may be out of range*. The verifier also knows that Go stops evaluating `&&` and
`||` early, so `i < len(A) && A[i] > 0` is fine with nothing said about `i`.

The `Ensures` shows that the prover knows what division means: the quotient and
the remainder add back up.

```go
//@ Requires n >= 0
func Halve(n int) (h int) {
    return n / 2
}
//@ Ensures 2*h <= n ^ n < 2*h + 2
```

## Loops

Nobody knows how many times a loop will go round, so the verifier doesn't
try to follow it. It asks for an **invariant** instead: a formula that is true
when the loop is reached, and true again after every round. It proves those two
things. After the loop, the invariant plus the loop condition being false is
everything it knows.

A **variant** is for termination: an integer expression that stays nonnegative
while the loop runs and shrinks every round. A loop without one is still proved
correct *if* it ends, and gets a warning saying so.

```go
//@ Requires n >= 0 ^ d > 0
func Divide(n, d int) (q, r int) {
    r = n
    //@ Invariant n = q*d + r ^ r >= 0
    //@ Variant r
    for r >= d {
        q, r = q+1, r-d
    }
    return q, r
}
//@ Ensures n = q*d + r ^ 0 <= r < d
```

Read the invariant against the `Ensures`. They differ only in `r < d`, and
that is exactly the loop condition being false. That's most of the secret to
finding invariants: take the postcondition and weaken it until it's true before
the loop starts.

`Requires d > 0` is there for the variant. With `d = 0` the loop runs forever,
and the verifier notices: *Variant is not shown to decrease*.

The two annotations can also sit at the end of the loop's body, which is where
VeGo's own examples tend to put them.

### Counted loops, `range`, `continue`

The counter of a `for` loop is in scope for the invariant.

```go
//@ Requires n >= 0
func SumTo(n int) (s int) {
    //@ Invariant 0 <= i <= n + 1 ^ 2*s = i*(i-1)
    //@ Variant n + 1 - i
    for i := 0; i <= n; i++ {
        s += i
    }
    return s
}
//@ Ensures 2*s = n*(n+1)
```

That's Gauss's formula, with the division multiplied away. Nothing here is
linear, and the proof goes through anyway, because `(i+1)*i` and `i*i + i` are
the same polynomial.

A `range` loop always ends, so it needs no variant. Its counter runs from `0`
to the length, and the verifier knows that without being told. `continue` jumps
to the end of the round, where the invariant is due as usual. The call to
`fmt.Println` is allowed because all it gets are numbers and a string, so it
can't change anything the proof cares about.

```go
func Count(A []int, x int) (n int) {
    //@ Invariant 0 <= n <= i
    for i, a := range A {
        if a != x {
            continue
        }
        n++
        fmt.Println("  found", x, "at", i)
    }
    return n
}
//@ Ensures 0 <= n <= |A|
```

### `break`, and what cannot happen

A `break` leaves the loop with whatever is known at that point. The invariant
isn't assumed there, because the round never finished.

`Exsures` is the negative of `Ensures`. It describes a state no return may ever
be in, which is sometimes the more natural thing to say: *it never happens that
a position is returned and the element there is not `x`*.

```go
func Find(A []int, x int) (at int) {
    at = -1
    //@ Invariant 0 <= i <= |A| ^ at = -1
    //@ Invariant x <> A[0:i)
    //@ Variant |A| - i
    for i := 0; i < len(A); i++ {
        if A[i] == x {
            at = i
            break
        }
    }
    return at
}
//@ Ensures at = -1 v 0 <= at < |A|
//@ Ensures at = -1 -> Forall k in [0, |A|) : A[k] <> x
//@ Exsures at >= 0 ^ A[at] <> x
```

## Quantifiers, ranges and predicates

`x <> A[0:i)` in the last example was a quantifier in disguise: *`x` differs
from every element of `A` from `0` up to but not including `i`*. The ends of a
range are written like the ends of an interval, `[` or `]` for an end that's
included and `(` or `)` for one that isn't. An order comparison with a range
holds for all of its elements. `=` and `in` hold for some element.

```go
//@ Requires |A| > 0
func Max(A []int) (m int) {
    m = A[0]
    //@ Invariant 1 <= i <= |A|
    //@ Invariant A[0:i) <= m ^ m in A[0:i)
    //@ Variant |A| - i
    for i := 1; i < len(A); i++ {
        if A[i] > m {
            m = A[i]
        }
    }
    return m
}
//@ Ensures A[:] <= m ^ m in A[:]
```

Both halves of that `Ensures` matter. `A[:] <= m` on its own is satisfied by
returning a billion. A contract that's too weak gets proved just as happily as
a good one, and it's on you, the reader, to notice.

A `Predicate` gives a formula a name. It isn't a function and never runs, using
it just means using its body. This one also shows a formula continuing on the
next line, after the `:`.

```go
//@ Predicate Sorted(A) ::= Forall i in [0, |A|) :
//@   Forall j in [i, |A|) : A[i] <= A[j]

//@ Predicate Between(lo, x, hi) ::= lo <= x ^ x <= hi
```

## Calls

Calls are where verification goes modular. The caller never looks inside the
callee. It proves the callee's `Requires`, and all it learns about the result
is the callee's `Ensures`. Nothing else.

```go
//@ Requires Between(0, x, 10)
func Twice(x int) (r int) {
    return x + x
}
//@ Ensures r = double(x) ^ Between(0, r, 20)

//@ Requires Between(1, k, 5)
func UseTwice(k int) (r int) {
    return Twice(k) + Twice(k+5)
}
//@ Ensures r = 4*k + 10
```

Change that to `Between(1, k, 6)` and the second call is refused: *this call
does not establish what Twice requires*.

`double` in that `Ensures` is a Go function. A function whose whole body is
`return` plus one expression can be used in a formula, where it means that
expression. It gives the contract a vocabulary it shares with the program.

```go
func double(x int) int { return 2 * x }
```

## Recursion

Recursion is induction. Inside `Triangle`, the call `Triangle(n-1)` is known
through the very contract being proved. That's the induction hypothesis, and
using it is sound as long as the induction is well-founded (it can't go down
forever). `Measure` is what makes it so: an expression that is nonnegative and
gets smaller at every recursive call. `BaseCase` marks where the induction
starts, and the verifier checks that no recursive call can be reached once it
has held. `InductionHypothesis` states, and proves, what makes the recursive
call legal.

```go
//@ Requires n >= 0
//@ Measure n
func Triangle(n int) (t int) {
    //@ BaseCase n = 0
    if n == 0 {
        return 0
    }
    //@ InductionHypothesis n - 1 >= 0
    return n + Triangle(n-1)
}
//@ Ensures 2*t = n*(n+1)
```

Write `Triangle(n)` instead and the contract would still "follow" from itself.
Circular reasoning, caught red-handed by the `Measure`: *Measure is not shown
to get smaller in the recursive call*.

## Assertions and primes

`Assert` is a claim about the state at one point. It's proved there and known
from there on, so it doubles as a stepping stone when the prover needs a hand.
The word is optional: a bare formula is an assertion.

A primed name is a value *after* an assignment. In a formula that mentions
`x'`, the `x'` is the value now and plain `x` is the value one assignment
earlier, and `x''` moves everything one step further back. Here's the old trick
of swapping two numbers without a third:

```go
func Swap(a, b int) (x, y int) {
    x, y = a, b
    x = x + y
    y = x - y
    x = x - y
    //@ Assert x = a + b ^ x' = b
    //@ y = a
    return x, y
}
//@ Ensures x = b ^ y = a
```

In an `Ensures`, a parameter without a prime is the value the caller passed,
whatever the body has done to its copy since.

## Promises that last: `Preserves`

`Preserves` states something once and gets it checked after every assignment
until the block ends. A loop inside that block may assume it, so the invariant
doesn't have to repeat it.

```go
//@ Requires amount >= 0
func Transfer(from, to, amount int) (f, t int) {
    f, t = from, to
    //@ Preserves f + t = from + to
    //@ Invariant 0 <= moved <= amount ^ f = from - moved
    //@ Variant amount - moved
    for moved := 0; moved < amount; moved++ {
        f, t = f-1, t+1
    }
    return f, t
}
//@ Ensures f + t = from + to ^ f = from - amount
```

No money is created or destroyed. The parallel assignment is the point: `f--`
followed by `t++` would break the promise for the length of one statement, and
the verifier would tell you which one.

## The escape hatch: `Axiom`

The prover is complete for nothing. Its arithmetic is linear, with products
treated as unknowns it knows a few facts about, and the cube of a number is
beyond it. `Axiom` tells it, and it believes you without proof. That is exactly
as dangerous as it sounds, so every `Axiom` gets reported as a warning, every
time.

```go
//@ Requires x >= 0
func Cube(x int) (c int) {
    //@ Axiom x*x*x >= 0
    return x * x * x
}
//@ Ensures c >= 0
```

## All of it at once: binary search

The most famous loop with a bug in it. The invariant says everything left of
`lo` is too small and everything from `hi` on is too big. `Sorted` is what lets
one comparison with `A[mid]` speak for a whole half of the slice.

```go
//@ Requires Sorted(A)
func Search(A []int, x int) (at int, found bool) {
    lo, hi := 0, len(A)
    //@ Invariant 0 <= lo <= hi <= |A|
    //@ Invariant A[0:lo) < x ^ x < A[hi:|A|)
    //@ Variant hi - lo
    for lo < hi {
        mid := lo + (hi-lo)/2
        if A[mid] == x {
            return mid, true
        } else if A[mid] < x {
            lo = mid + 1
        } else {
            hi = mid
        }
    }
    return 0, false
}
//@ Ensures found -> 0 <= at < |A| ^ A[at] = x
//@ Ensures ~found -> x <> A[:]
```

Four things are proved here, and only two were written down. If `found`, then
`at` is a position of `x`. If not, `x` is nowhere in `A`. And, unasked,
`A[mid]` is always inside the slice, and the loop ends. Now break it. Try
`lo = mid` (it loops forever, and the `Variant` says so), `hi = mid - 1` (it
skips an element, and the invariant isn't maintained), or drop the `Requires`
(and nothing about the halves follows).

## What is left out

litgo's checker is its own, because the paper's `vegop` hasn't been
published, and it's deliberately smaller than what the paper describes.

* **The fragment.** Integers, booleans, slices of integers that are read and
  not written, integer and boolean fields of structs, `if`, `for`, `range`,
  `break`, `continue`, `return`, and calls of annotated functions. An
  annotated function that uses anything else is reported as *not verified*,
  with the line that went too far. It's never silently passed.
* **No writes through a slice or a pointer.** `A[i] = x` might change another
  slice that shares the array, and a proof that didn't notice would prove
  something false. So sorting is out, for now.
* **Not Go, so not here.** VeGo's `.vgo` dialect has `while`, `skip` and
  functions inside functions. A litgo block is Go, where `for cond {}`, an
  empty statement and a second function say the same things.
* **Read but not checked.** `Property`, and contracts on interface methods.
  They get a warning, not silence.
* **The integers are mathematical.** Overflow isn't modelled, here or in the
  paper.
* **"Not proved" is not "false".** The prover never calls a false thing
  proved, but sometimes it fails to prove a true one. The cure is usually a
  stronger invariant, or an `Assert` along the way.

## Running it

```go
func main() {
    fmt.Println("AddPositives(2, 3) =", AddPositives(2, 3))
    fmt.Println("ElementOver({7, 9}, 1, 2) =", ElementOver([]int{7, 9}, 1, 2))
    fmt.Println("Halve(7) =", Halve(7))
    q, r := Divide(17, 5)
    fmt.Println("Divide(17, 5) =", q, r)
    fmt.Println("SumTo(100) =", SumTo(100))
    A := []int{1, 3, 3, 7, 9, 12}
    fmt.Println("Count(A, 3):")
    fmt.Println("  =", Count(A, 3))
    fmt.Println("Find(A, 7) =", Find(A, 7))
    fmt.Println("Max(A) =", Max(A))
    fmt.Println("UseTwice(2) =", UseTwice(2))
    fmt.Println("Triangle(10) =", Triangle(10))
    x, y := Swap(1, 2)
    fmt.Println("Swap(1, 2) =", x, y)
    f, t := Transfer(10, 0, 4)
    fmt.Println("Transfer(10, 0, 4) =", f, t)
    fmt.Println("Cube(3) =", Cube(3))
    at, found := Search(A, 9)
    fmt.Println("Search(A, 9) =", at, found)
}
```

An example with a tree in it

The fourth example goes after data structures: a binary heap (a tree kept in a slice) and a ring buffer. The heap’s numbering really is a tree. The heap property really does put the maximum at the root. The ring’s operations really do keep its invariant. Not “the tests pass”. Proved.

examples/heap.lit.md
<!-- tangler: go -->
<!-- package: main -->
<!-- imports: fmt -->

# A tree in an array, and a queue in a circle

A binary heap is a tree with no pointers in it. A ring buffer is a queue with
no end. Both are nothing but integers and a slice, which makes them perfect
first subjects for a proof.

Everything here is proved by `litgo prove`, in the notation of
[VeGo](https://arxiv.org/abs/2608.22630). For a data structure, "proved" comes
in three flavors, and you'll meet all of them below:

* **The shape is right.** The arithmetic that finds a parent or a child really
  does describe a tree. Children know their parent, nobody is their own
  ancestor, and no index falls off the end of the slice.
* **The invariant means something.** "Every node is at most its parent" is a
  local fact. "The root is the maximum" is a global one. Getting from one to
  the other is an induction, and in code, an induction is a loop.
* **The operations keep the invariant.** If a ring buffer was valid before
  `Push`, it is valid after. Whatever state it was in.

```mermaid
flowchart TD
    n0["A[0] = 9"] --> n1["A[1] = 7"] & n2["A[2] = 8"]
    n1 --> n3["A[3] = 3"] & n4["A[4] = 5"]
    n2 --> n5["A[5] = 6"]
```

## The shape of the tree

There are no pointers. The tree lives in the numbering. Node $i$ has its
children at $2i+1$ and $2i+2$, so its parent is at $\lfloor (i-1)/2 \rfloor$.
That's three one-line functions, and their contracts say they fit together.

```go
//@ Requires i >= 0
func Left(i int) (c int) { return 2*i + 1 }
//@ Ensures Parent(c) = i ^ c > i

//@ Requires i >= 0
func Right(i int) (c int) { return 2*i + 2 }
//@ Ensures Parent(c) = i ^ c = Left(i) + 1

//@ Requires i > 0
func Parent(i int) (p int) { return (i - 1) / 2 }
//@ Ensures 0 <= p < i
//@ Ensures i = Left(p) v i = Right(p)
```

For `Parent(Left(i)) = i`, the verifier has to know that $2i/2 = i$ and also
that $(2i+1)/2 = i$ (that's what rounding down means). A function whose whole
body is one `return` of one expression can be used inside a formula, where it
stands for that expression. That's how the contracts get to talk about each
other.

`0 <= p < i` is the one that matters. A parent's number is smaller than its
child's. So every step upward makes progress, there are no cycles, and every
walk ends at node 0. That is what makes this numbering a *tree*.

## The heap property

A max-heap is a tree where no node is larger than its parent. Here it is as a
predicate over the whole slice:

```go
//@ Predicate Heap(A) ::= Forall k in [1, |A|) : A[Parent(k)] >= A[k]
```

Checking it is a loop over every node that has a parent. The contract is an
*if and only if*. `true` means it's a heap, and `false` means it isn't, which
is the half people forget to promise.

```go
func IsHeap(A []int) (ok bool) {
    //@ Invariant 1 <= i
    //@ Invariant Forall k in [1, i) : k < |A| -> A[Parent(k)] >= A[k]
    //@ Variant |A| - i
    for i := 1; i < len(A); i++ {
        if A[Parent(i)] < A[i] {
            return false
        }
    }
    return true
}
//@ Ensures ok <-> Heap(A)
```

Notice what nobody wrote: that `A[Parent(i)]` is inside the slice. The
verifier works that out from `1 <= i < len(A)` and the meaning of division. Or
it refuses the function.

## From local to global: the root is the maximum

The heap property only ever compares a node with its parent. So how do we know
the root beats *every* node? By climbing. If $A[j] \ge A[i]$ and $j$ is not
the root, then $A[\mathrm{Parent}(j)] \ge A[j] \ge A[i]$, and the parent is one
step closer to the top.

```go
//@ Requires Heap(A) ^ 0 <= i < |A|
func Depth(A []int, i int) (d int) {
    j := i
    //@ Invariant 0 <= j <= i ^ A[j] >= A[i] ^ d >= 0
    //@ Variant j
    for j > 0 {
        j = Parent(j)
        d++
    }
    return d
}
//@ Ensures d >= 0 ^ A[0] >= A[i]
```

The function counts how deep node `i` is, but the proof is the interesting
part. The invariant carries `A[j] >= A[i]` up the tree like a torch, and when
the loop ends, `j` is 0. The `Variant` is `j` itself, which shrinks because
`Parent(j) < j`. One contract proves the climb is correct *and* that it ends.

That was one node. For all of them at once, the induction runs over the
numbering instead of up a path. If the root beats every node below `k`, it
beats `Parent(k)`, which beats `k`.

```go
//@ Requires Heap(A) ^ |A| > 0
func Max(A []int) (m int) {
    //@ Invariant 1 <= k <= |A| ^ A[0:k) <= A[0]
    //@ Variant |A| - k
    for k := 1; k < len(A); k++ {
        //@ Assert A[Parent(k)] <= A[0]
    }
    return A[0]
}
//@ Ensures A[:] <= m ^ m in A[:]
```

This loop does nothing. (The `Assert` is the step of the induction, written
out for you. The prover finds it without being told.) The loop is there
because the loop *is* the proof. VeGo has no
ghost code (code that exists only for the proof and never runs), so an
induction has to be a loop that could run, and this one runs. It costs a whole
pass over the slice to return `A[0]`, which no real heap would pay, so the
honest way to use it is as a lemma, once, in a test. It's worth seeing anyway,
because it shows what a loop invariant *is*: the induction hypothesis, with the
loop as the induction.

## One step of sifting down

You repair a heap by swapping a node with its larger child. Picking that child
is where the off-by-one errors live, because a node can have two children, or
one, or none.

```go
//@ Requires i >= 0 ^ Left(i) < |A|
func LargerChild(A []int, i int) (c int) {
    c = Left(i)
    if Right(i) < len(A) && A[Right(i)] > A[c] {
        c = Right(i)
    }
    return c
}
//@ Ensures (c = Left(i) v c = Right(i)) ^ c < |A| ^ Parent(c) = i
//@ Ensures A[c] >= A[Left(i)] ^ (Right(i) < |A| -> A[c] >= A[Right(i)])
```

Go stops evaluating `&&` at the first `false`, and the verifier knows it. So
`A[Right(i)]` only has to be in range once `Right(i) < len(A)` has just been
checked. Swap the two halves of that condition and the proof fails, right at
the index.

The swap itself is `A[i], A[c] = A[c], A[i]`, and that's where this verifier
stops. It doesn't follow writes into a slice, because another slice may share
the same array. So `main` below does the swapping, unproved, with the proved
function telling it where.

## A queue in a circle

A ring buffer is three integers and a slice: where the oldest element sits, how
many elements there are, and how much room there is. The invariant is what
turns those three numbers into a queue.

```go
//@ Predicate Ring(head, size, room) ::= room > 0 ^ 0 <= head < room ^ 0 <= size <= room
```

Every operation has the same contract, in outline: *give me a valid ring, and
I'll give you a slot inside the buffer and a valid ring back*. The parameters
are integers passed by value, so they can't change, and the new state comes
back as results.

```go
//@ Requires Ring(head, size, room) ^ size < room
func Push(head, size, room int) (slot, h, s int) {
    return (head + size) % room, head, size + 1
}
//@ Ensures 0 <= slot < room ^ Ring(h, s, room) ^ s = size + 1

//@ Requires Ring(head, size, room) ^ size > 0
func Pop(head, size, room int) (slot, h, s int) {
    return head, (head + 1) % room, size - 1
}
//@ Ensures 0 <= slot < room ^ Ring(h, s, room) ^ s = size - 1
```

`size < room` and `size > 0` are the two ways to misuse a queue (push onto a
full one, pop from an empty one). Now they are the caller's problem, in
writing. A caller that is verified too has to prove them:

```go
//@ Requires Ring(head, size, room) ^ size < room
func PushThenPop(head, size, room int) (h, s int) {
    _, h, s = Push(head, size, room)
    _, h, s = Pop(h, s, room)
    return h, s
}
//@ Ensures Ring(h, s, room) ^ s = size
```

`Pop` is legal there because `Push` promised `s = size + 1`, which is more
than zero. Delete that promise from `Push`'s contract and this function stops
verifying, even though `Push` itself still does. A contract is everything a
caller gets to know.

## Running it

```go
func main() {
    A := []int{9, 7, 8, 3, 5, 6}
    fmt.Println("IsHeap:", IsHeap(A), " Max:", Max(A), " Depth of node 5:", Depth(A, 5))

    // Replace the root and sift it down, one proved step at a time.
    A[0] = 1
    for i := 0; Left(i) < len(A); {
        c := LargerChild(A, i)
        if A[i] >= A[c] {
            break
        }
        A[i], A[c] = A[c], A[i]
        i = c
    }
    fmt.Println("after sifting 1 down:", A, " IsHeap:", IsHeap(A))

    buf := make([]int, 3)
    head, size := 0, 0
    for _, x := range []int{10, 20, 30} {
        var slot int
        slot, head, size = Push(head, size, len(buf))
        buf[slot] = x
    }
    var slot int
    slot, head, size = Pop(head, size, len(buf))
    fmt.Println("popped", buf[slot])
    slot, head, size = Push(head, size, len(buf))
    buf[slot] = 40
    fmt.Println("buffer:", buf, " head:", head, " size:", size)
}
```

## Things to break

A proof you've never seen fail is hard to trust. So go break things:

* In `Parent`, return `i / 2`. `Left` still verifies. `Right` doesn't, and
  neither does `Parent` itself, or `LargerChild`. The tree has lost its right
  children, and everything that relied on them says so.
* In `Depth`, climb with `j = j - 1`. It still ends, but the invariant breaks,
  because the node before `j` is no relation of it.
* In `LargerChild`, test `A[Right(i)] > A[c]` first. The index is no longer
  known to be in range.
* In `Push`, drop the `% room`. The slot escapes the buffer.

Publishing

The last two files aren’t part of litgo. They’re what GitHub does with litgo after a push. One publishes this document as a web page, and the other publishes the binaries.

.github/workflows/pages.yml

This document is also the project’s web page. Every push to master builds litgo from the committed sources, weaves this file to HTML with the binary it just built, and publishes the result to GitHub Pages. The examples go up beside it. Every examples/*.lit.md becomes examples/<name>.html on the same site, so worker_pool, isqrt, vego-tutorial and heap can be read without cloning anything, and a new example goes live the moment it’s committed. The tests run first, and so do the proofs (of this document and of the examples), so a push that breaks litgo never replaces a page that works. The HTML needs only pandoc, because the browser runs mermaid.js and KaTeX itself. pandoc is pinned to a release, since the one Ubuntu packages is years older, and the page should look the way it does when woven at home.

name: pages

on:
  push:
    branches: [master]
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

concurrency:
  group: pages
  cancel-in-progress: true

env:
  PANDOC: 3.10.2

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - name: Install pandoc
        run: |
          curl -fsSL -o pandoc.deb "https://github.com/jgm/pandoc/releases/download/$PANDOC/pandoc-$PANDOC-1-amd64.deb"
          sudo dpkg -i pandoc.deb
      - name: Build, test and prove
        run: |
          go vet ./...
          go test ./...
          go build -o bin/litgo .
          bin/litgo prove litgo.lit.md examples/isqrt.lit.md examples/vego-tutorial.lit.md examples/heap.lit.md
      - name: Weave
        run: |
          mkdir -p _site/examples
          bin/litgo weave --to html -o _site/index.html litgo.lit.md
          for f in examples/*.lit.md; do
            name=$(basename "$f" .lit.md)
            bin/litgo weave --to html -o "_site/examples/$name.html" "$f"
          done
      - uses: actions/upload-pages-artifact@v3
        with:
          path: _site

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - id: deployment
        uses: actions/deploy-pages@v4
.github/workflows/release.yml

Releases follow the VERSION constant in the command. Every push to master reads it, and if nothing is tagged v plus that version yet, the push becomes that release. Publishing a version means changing one string. A push that leaves the string alone publishes nothing. Because litgo depends on nothing but the standard library, cross-compiling is a matter of setting two variables, and one Linux machine builds for all four targets. The tests run first here too.

This is the last block. Tangle the document it sits in and you get litgo. Run that litgo on this document and you get it all back, byte for byte, including this block. The snake eats its tail one more time and comes out the same length.

name: release

on:
  push:
    branches: [master]
  workflow_dispatch:

permissions:
  contents: write

concurrency:
  group: release

jobs:
  release:
    runs-on: ubuntu-latest
    env:
      GH_TOKEN: ${{ github.token }}
    steps:
      - uses: actions/checkout@v4
      - name: Is this version released?
        id: version
        run: |
          version=$(sed -n 's/^const VERSION = "\(.*\)"$/\1/p' main.go)
          test -n "$version"
          echo "tag=v$version" >> "$GITHUB_OUTPUT"
          if gh release view "v$version" > /dev/null 2>&1; then
            echo "v$version is already released"
          else
            echo "new=true" >> "$GITHUB_OUTPUT"
          fi
      - uses: actions/setup-go@v5
        if: steps.version.outputs.new
        with:
          go-version-file: go.mod
      - name: Test
        if: steps.version.outputs.new
        run: |
          go vet ./...
          go test ./...
      - name: Build
        if: steps.version.outputs.new
        run: |
          tag=${{ steps.version.outputs.tag }}
          mkdir dist
          for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do
            os=${target%/*} arch=${target#*/}
            dir=litgo_${tag#v}_${os}_${arch}
            mkdir "$dir"
            CGO_ENABLED=0 GOOS=$os GOARCH=$arch go build -trimpath -ldflags "-s -w" -o "$dir/litgo" .
            cp LICENSE README.md "$dir"
            tar -czf "dist/$dir.tar.gz" "$dir"
          done
          (cd dist && sha256sum *.tar.gz > checksums.txt)
      - name: Publish
        if: steps.version.outputs.new
        run: |
          gh release create ${{ steps.version.outputs.tag }} dist/* \
            --target "$GITHUB_SHA" --title ${{ steps.version.outputs.tag }} --generate-notes

litgo

func main() {
    fmt.Println("Hello from litgo!")
}