Skip to content

Latest commit

 

History

History
673 lines (532 loc) · 66.6 KB

File metadata and controls

673 lines (532 loc) · 66.6 KB

mdreflow design

2026-08-07. Status: M0–M5 implemented — all modes, full skip-list, CLI, and release packaging (goreleaser, pre-commit hooks); typography shipped in M5 and was later removed (see Typography). This is a living document: design changes land here first, then in code.

mdreflow is a Go library and CLI that reflows Markdown prose. Its home mode is sentence-per-line (semantic line breaks), with paragraph-per-line and hard-wrap modes sharing the same pipeline. It is a reflow tool, not a formatter: it changes where lines break inside paragraph prose and touches nothing else. why-this-is-hard.md is the companion rationale: the block/span/cluster model, the reparse ambiguities that force the freeze-based design here, and what authors can do about a frozen paragraph.

  • Repo/module/package/binary: github.com/jbeda/mdreflow / mdreflow
  • License: Apache-2.0

Motivation

Sentence-per-line Markdown makes diffs reviewable (one sentence changed → one line changed) and gives agents and humans a stable convention for prose edits. Existing tools have gaps: flowmark misses sentence boundaries adjacent to inline formatting (#68) and bypasses excludes for explicitly named files (#43); mdslw and rumdl are solid but Rust and CLI-only. None is embeddable as a Go library, which is what other Go tooling needs to format Markdown it generates or manages.

Agents are a primary driver: they generate and edit a lot of Markdown, they run the tool unattended, and they occasionally point it at the wrong file. The design favors loud, machine-legible behavior (exit codes, refusals, complete --help) over interactive polish.

Goals and non-goals

Goals:

  • Reflow paragraph prose in three modes: sentence-per-line, paragraph-per-line, hard wrap.
  • Be safe to run unattended: idempotent, render-preserving, loud about anything surprising.
  • Work as a Go library first; the CLI is a thin wrapper.
  • Handle real-world dialect content (GFM, MDX/Docusaurus, Hugo) by skipping it, not by parsing it.

Non-goals — these are hard principles, and changing them means amending this document first:

  • No normalization. mdreflow never rewrites block structure: no heading style changes, list marker unification, table alignment, reference-link consolidation, or escaping changes. Tools like rumdl do that tier well and compose cleanly with mdreflow because the two touch disjoint constructs.
  • No Markdown renderer. The Markdown parser is used read-only to locate prose; output is produced by splicing reflowed prose into verbatim source bytes. This is what keeps the correctness surface small; the multi-year bug tails in normalizing formatters (Prettier's proseWrap, mdformat's escaping engine) live on the other side of this line.

Explicitly deferred or unlikely (see Future directions): formatting Markdown embedded in YAML/JSON values, per-file options in front matter, a --dialect selector, content-based file-type sniffing.

Modes

Mode Behavior
sentence (default) Join each paragraph's lines, split at sentence boundaries, one sentence per line. With --max-width N, sentences longer than N get secondary breaks at clause boundaries (commas, semicolons), falling back to the last word boundary. Without it, long sentences stay long.
para Join each paragraph to a single line.
wrap Classic hard wrap at --max-width (default 80), breaking at word boundaries only.

All modes are one pipeline — join lines, compute break points, emit — differing only in the break-point strategy. Continuation lines inside list items and blockquotes are re-indented to match their container.

Width-constrained modes measure candidate lines against the cluster-global no-break spans via a per-cluster prefix table (widthMeasurer in internal/reflow), built once and queried in O(1); the escape simulation (escapeBlockInterrupt) is evaluated exactly only inside the narrow width band where it can change the verdict. An earlier implementation re-parsed every candidate prefix from scratch, which was roughly cubic (about 30 seconds for a 2000-word paragraph); the rewrite was verified against it by a full-corpus differential run (fixtures plus all fuzz seeds, byte-identical) and the truncated-prefix vs. global-span measurement question is documented on widthMeasurer. BenchmarkFormat pins the scaling; width-constrained modes must stay near-linear in paragraph size.

Architecture

source bytes
   │
   ├─ goldmark parse (read-only) ──► block map: paragraph byte-ranges,
   │                                 skip-list ranges, inline no-break spans
   │
   ├─ for each paragraph: join lines → find break points → re-emit
   │
   └─ everything else: emitted byte-for-byte

goldmark (with its GFM extensions enabled for recognition purposes) parses the document; its AST carries byte offsets into the source, which is all we need. From the AST we derive:

  • Paragraph ranges: the blocks eligible for reflow, with their container context (list item, blockquote) for continuation indentation.
  • No-break inline spans within paragraphs: inline code, links and link destinations, autolinks, images, inline math, footnote references, inline HTML/JSX. Break points never land inside these; a span longer than any width limit simply overflows. These are computed on the joined cluster text (which exists nowhere in the document), by parsing that text through goldmark itself — see "No-break spans: ask goldmark" below.
  • Skip ranges: everything below.

Convergence: reflow runs to fixpoint

A single pipeline pass plans breaks from the pre-reflow parse, but the guarantees are judged on the post-reflow reparse. Fuzzing showed a persistent family of corners where the two disagree: the emitted text parses slightly differently than the planner assumed (an escape changes code-span pairing, a consumed double space changes a sentence-boundary verdict, a join changes a link-reference-definition skip decision), so a second run lands differently than the first.

The goal is single-pass convergence: the planner should predict its own output's reparse correctly the first time, and every known divergence gets a root-cause fix. But unknown shapes will keep existing, so Format also makes idempotency structural as a backstop: it runs the pipeline, then re-runs it on its own output until the output is stable, up to a small cap (4 passes; in practice the first re-run already matches). If the cap is hit without convergence — including a cycle where two outputs alternate — Format returns the original document unchanged. Document-level, not per-paragraph: the known divergence modes merge or re-split paragraphs across passes, so "the same paragraph, one pass later" is not a stable identity to fall back on. "We could not safely flow this" is expressed as a no-op, never as churn.

The backstop is for users; internally it is treated as a bug detector. The test and fuzz harnesses disable it (test-only switch) and drive the core directly, so an input that needs the backstop shows up as a failing idempotency oracle to be root-caused rather than silently absorbed.

What the oracle requires is that the core settles, within a bound strictly tighter than the backstop's own (coreConvergePasses against maxFormatPasses), not that it settles on the first pass. A planner bug that never converges, or cycles between two outputs, still fails the oracle — that is the shape the backstop would otherwise paper over by returning the document unformatted. Single-pass convergence remains the goal and the default disposition for any divergence; where a root-cause fix would cost more reflow coverage than the divergence costs, the divergence is documented instead, and the gate soak reports the share of inputs that needed a second pass so the class stays visible as a number that can be watched across a change. See "Emphasis openers are allowed to cost a second pass" under Sentence segmentation for the one such exception and the priority argument behind it.

Render preservation is also structural: the render backstop

Idempotency's twin guarantee gets the same treatment. (Amendment 2026-08-09; supersedes the earlier stance that render hazards were planner obligations only, with the fuzz oracle as sole referee.) After the fixpoint loop settles, Format renders the input and the candidate output through the same goldmark configuration the parse used (see Dialects) and compares normalized HTML — the same normalization the fuzz render oracle applies, promoted from test code to an internal package. On any difference, Format returns the original document unchanged — document-level, for the same reasons as the idempotency fallback, and expressed the same way: "we could not safely flow this" is a no-op, never corruption.

This inverts the safety architecture. The blockmap guards (spanning-delimiter checks, line-start hazard filtering, definition zones) stop being the mechanism that makes render preservation true and become a coverage mechanism: their job is maximizing how much prose reflows without tripping the backstop, and an incomplete guard now costs a missed reflow instead of corrupted content. Guard narrowing becomes correspondingly lower-stakes.

One carve-out, for dialect recognitions that reinterpret a construct: under the mkdocs dialect an admonition body is prose to the target renderer but an indented code block to goldmark, so reflowing it changes goldmark's render by design, and a naive comparison would veto every such reflow. The backstop therefore checks each dialect-recognized reflow-eligible range against its own invariant — the word sequence is unchanged, only inter-word whitespace moved (for an indented block, goldmark's <pre> output preserves line breaks, so whitespace-insensitive comparison of that block's rendered text is exactly this check) — and the whole-document comparison normalizes those ranges' rendered content identically on both sides. Everything outside a recognized range keeps the exact comparison. This is not a loosening in disguise: "words unchanged, whitespace moved" is the same promise reflow makes for an ordinary paragraph; it is simply enforced directly where goldmark's semantics cannot express it.

What the backstop measures is self-consistency under goldmark's semantics: input and output pass through the same parser, so goldmark's own bugs largely cancel — the comparison still detects whether reflow changed anything under those semantics. What it cannot measure is goldmark-vs-target-renderer divergence (Python-Markdown above all; cmark-gfm marginally; Hugo not at all, Hugo is goldmark). That exposure predates the backstop — the planner has always used goldmark's AST as its parse authority — and is bounded by conservative per-dialect recognition and external verification (site-build diffs), not by this check.

Same discipline as the idempotency backstop: it is for users, internally a bug detector. The harnesses disable it and assert render preservation as a failing oracle instead, so every shape that would trip it gets root-caused. A false trip — the backstop suppressing a legitimate reflow of ordinary prose, most plausibly through an over-strict normalization — is itself a bug class, watched by running the corpus with the backstop enabled and asserting zero fallbacks.

Two things the backstops deliberately do not do:

  • They do not excuse planner bugs. Every hazard remains a planner obligation fixed at the root, with the fuzz oracles as referee; the backstops only bound the blast radius of the shapes nobody has found yet.
  • They are not a license for sloppy single-pass behavior. Each known divergence shape gets a root-cause fix; the backstops turn unknown shapes from correctness bugs into (at worst) unreflowed documents.

Cost: the common case is exactly two pipeline passes (one to reflow, one to confirm stability) plus two parse-and-renders for the comparison. Post the width-measurement fix a pass is milliseconds even on large documents; the benchmark suite pins this and grows a render-backstop benchmark alongside.

Dialects: renderer profiles, and the skip-list

(Amendment 2026-08-09, with PR #19 as the first consumer.) Options.Dialect — a single-select enum mirroring Mode's shape, with --dialect and a dialect: config key parsed loudly like mode — names the renderer profile a tree targets. A dialect is a bundle, not a feature flag. It selects:

  1. The goldmark configuration used for parsing and for the render backstop's comparison, so "render-preserving" always means "under the target's semantics."
  2. Which flavour-specific block recognitions are on. Internally each recognition is a plain feature bit (reflow.Options.MkDocs is the first); dialects bundle bits at the Format boundary, and the bits themselves are never user-visible. If finer-grained selection is ever needed, the bits are already there — dialects would become named bundles over them — but single-select is the bet that we won't need it.
  3. Eventually, which of the skip-list rules below even apply — a construct the profile's parser cannot produce stops being a hazard automatically under the render backstop: with tables off, a manufactured delimiter-shaped line is prose on both sides of the comparison. Over-cautious guards therefore cost coverage, never correctness, and can be narrowed per-dialect lazily or not at all.

The zero value, DialectGFM (--dialect gfm), is exactly the permissive GFM-plus-footnotes configuration the parser has always hardcoded — existing behavior, renamed rather than changed. commonmark is deliberately reserved for a future strict CommonMark profile (GFM extensions off) rather than aliased to the default: aliasing it now would make the name unavailable for the one profile it accurately describes. mkdocs layers admonition-body recognition on the GFM base and drops the linkify extension from its goldmark configuration: Python-Markdown has no GFM autolinking, so a bare URL is plain prose to the target — and because the span computation and the render backstop both use the profile's configuration (see "No-break spans: ask goldmark"), bare URLs automatically stop being protected or compared as links under this dialect, with no per-dialect guard logic anywhere. Dropping linkify cannot let reflow split a URL in any mode: every break candidate is a whitespace run, and a bare URL contains none, so reflow only ever moves the whitespace around the token — which autolinks (or doesn't) identically at line start, mid-line, or line end. The only behavior linkify actually governs in span computation is code-span pairing around a backtick inside a bare URL, a shape near-nonexistent in real prose. Its true target renderer (Python-Markdown) is not CommonMark and cannot be modeled by our oracle, so its recognitions must stay narrow and are verified externally (full mkdocs build diffs), per the render-backstop section's divergence caveat.

(Amendment 2026-08-11.) The admonition-marker line itself — the trigger for the body recognition above, and for the same line acting as an immovable boundary when a marker and its body are lazily joined into one paragraph (no blank line between them) — is recognized from a bounded prefix: !!! or ???/???+, whitespace, then at least one more byte. Nothing about the verdict depends on where the line ends, and nothing constrains what follows the class word, so Material for MkDocs's inline modifiers (!!! note inline end "Title") and non-alphabetic types both match. The class word is required: a line that is only the punctuation, or runs it straight into other characters (!!!bang, !!!!x, ???01010), is not a marker and must not claim the indented block below it — where that block is an indented code block, treating it as admonition prose reflows and escapes its contents, changing what the document renders to. This recognition, and the boundary it produces, are mkdocs-only; under gfm a line starting !!! or ??? is ordinary prose. See why-this-is-hard.md's "Why the verdicts are blunt" for why the prefix match replaced an earlier, end-anchored, type-word-requiring regex.

(Amendment 2026-08-12.) The two halves of that verdict answer to different requirements and are now separated. A paragraph's first line is pinned — never joined onto — whenever it opens with the marker punctuation at all, class word or not; only a line that also carries a class word is a marker whose body takes the 4-space indent. Pinning on the punctuation alone is what makes the verdict stable across passes: whether a paragraph is the lazily-joined admonition form is read from its first line, and reflow can rewrite that line, so wrapping a bare !!! used to pull the next word up and manufacture !!! word — which the following pass read as a marker and indented the body under, output that never settled. The first line's opening bytes are the one part of a paragraph reflow cannot move, so keying on them is verdict-stable; keeping the indent gated on a real class word preserves the rule above, since indenting under a bare !!! would turn ordinary prose into an indented code block.

An admonition body is left alone in two further cases, both because reflow would otherwise change what a CommonMark parser reads out of the block it sees:

  • Indented past 4 spaces. Reflow re-emits every body line at exactly the 4-space indent, so the extra whitespace — content, to the indented code block goldmark reads there — would be deleted. Bodies worth reflowing sit flush at 4.
  • An escape would be needed. A body is prose to MkDocs and code to CommonMark, so a backslash reflow adds to stop a wrapped line reparsing as a list or a fence is markup to the first renderer and literal text to the second; no single output preserves both renders. The paragraph backs out to its source bytes the moment an escape proves necessary (blockmap.Paragraph.EscapeIsContent), rather than teaching the render oracle to stop looking at admonition bodies — coverage is the cheaper thing to spend here than sight in the one area goldmark is already blind.

For everything else mdreflow still targets one permissive superset of dialects ("do our best on everything"). Dialect awareness beyond the profile is a skip-list — constructs recognized only well enough to pass through untouched — not a set of dialect implementations. Each rule is tagged by origin so dialect profiles can subset the existing rules rather than require new machinery:

Construct Dialects Handling
Fenced/indented code blocks CommonMark skip block
YAML/TOML front matter GFM tooling, Hugo, Docusaurus skip block (byte-for-byte)
Tables GFM skip block
Raw HTML blocks CommonMark skip block
JSX blocks, {expr} spans MDX skip block / no-break span
::: directives Docusaurus/remark skip fence lines; interior prose reflows
Math blocks ($$) GFM, Docusaurus skip block
Hugo shortcodes ({{< … >}}, {{% … %}}) Hugo skip block / no-break span
GitHub alert first-lines (> [!NOTE]) GFM marker line is an immovable boundary (it fuses into the quote's paragraph in the AST — see M0 findings); rest of quote reflows
Link reference definitions CommonMark skip zone (below)
Footnote definitions ([^label]:) GFM body reflows; continuations indented (below)
Hard line breaks CommonMark immovable boundary (below)

Spike M0 outcome (details and evidence in m0-spike-findings.md): goldmark does not parse MDX, and many dialect constructs do land as Paragraph nodes — but a general line-scanning pre-pass is not needed. The block-map derivation step grows content-pattern rules instead: (1) skip whole Paragraph nodes whose text matches a marker regex (::: fences, block shortcodes, block {expr}, +++ TOML front matter); (2) within a Paragraph, treat any Lines() segment matching a marker regex as an immovable boundary — this handles fences fused with prose by lazy continuation (no-blank-line :::, > [!NOTE] markers, paired shortcodes, $$ math); (3) recognize YAML front matter ourselves via a byte-range pre-scan (originally goldmark-meta, replaced — see Dependencies); (4) inline {expr}/shortcodes/math are invisible to the AST (plain Text) and need the already-planned regex scan for no-break spans. The one construct post-parse rules cannot recover is a multi-line JSX opening tag whose closing > sits alone on a line (goldmark consumes it as an empty blockquote) — documented as a known limitation, revisit with a narrow raw-line fence if it shows up in real content.

Sentence segmentation

Regex/punctuation splitting plus an abbreviation exception list — the approach every shipping tool uses (flowmark, mdslw, rumdl), and the approach whose known failure modes are documented in their issue trackers, which double as our test spec. The flowmark #68 lesson is baked in: sentence-terminal punctuation must be recognized through trailing inline markup (`code`., **bold**., quotes, parens).

A sentence may also open with inline markup (amendment 2026-08-12). The mirror of the flowmark #68 case: a sentence beginning with a code span or an emphasis run is a sentence start, so `code` opens the second one. splits from the sentence before it exactly as a plain word would. The character allow-set that decides this — uppercase, digit, opening quote, opening bracket — cannot express the rule, because the deciding fact is what the following bytes parse as, not which byte leads them. Adding `, * and _ to the set is the wrong fix twice over: it accepts a backtick that opens no code span at all, and it accepts one whose delimiter run is three or more, which becomes a fenced-code-block opener the moment the break puts it at a line start.

So the opener is AST-confirmed, the same doctrine no-break spans and hard-break detection already follow: a leading `, * or _ is a cheap byte-level prefilter, and the candidate is only accepted when the gm.NewInline() parse reports a code span or emphasis node beginning at exactly that offset. An unmatched delimiter parses as plain text and is correctly refused. The lowercase-word case keeps joining, which is what keeps e.g. foo intact.

Fence-shaped openers are refused before they are ever proposed, not after. The guard against a break landing immediately before a fenced-code-block opener shape moves from filterUnsafeLineEnds, which only width-based cuts reach, to filterLineStartHazards, which every candidate reaches — a following-line hazard belongs with the other following-line hazards. Leaving it where it was would have let the render backstop catch the fence instead, and that is a bad outcome rather than a safe one: the backstop reverts the whole file and returns src, so Format(src) == src and --check reports clean while every unrelated reflow in that file is silently dropped. Refusing the candidate keeps the failure in the one paragraph that earned it.

* and _ need no such guard at the landing position: a list marker requires a following space and a thematic break requires a line of nothing but markers and whitespace, while emphasis requires non-whitespace immediately after the opening delimiter, so an emphasis run at a line start can be neither.

Emphasis openers are allowed to cost a second pass (amendment 2026-08-12). Emphasis pairing is not a local fact. CommonMark decides whether a delimiter can open or close from the characters on either side of its run, so a run with whitespace on both sides is inert — it can neither open nor close, and every other delimiter in the paragraph is left partnerless. Strand that run alone on a line and escapeBlockInterrupt escapes it as a thematic break, which puts a backslash against it; the run now has punctuation on its left and a line ending on its right, which makes it a legal closer, which retroactively promotes a distant delimiter into a real emphasis opener. Pass 1 sees no opener and pass 2 sees one. Found by the fuzz gate on ! _!___0 __________ and 0!x99! **0yy! ~~* ** *, both at ModeSentence with a width, where reflow's own cut strands and escapes a run.

This is the one place the pipeline deliberately declines to converge in a single pass. Two guards were tried and both were withdrawn: refusing an opener whenever the cluster held three or more */_, and refusing only when such a run was strandable (whitespace on both sides, looking through a backslash). The first silently disabled the fix for any paragraph containing ***bold italic***. The second missed that a thematic break may space its markers out, so ** * is three markers and not a two-run, and the fuzzer found the same instability through that shape within one soak. Each guard predicted where escaping would land; the escape sites are thematic breaks, fence openers, bullets and setext underlines, and any of them can put a backslash against an emphasis delimiter, so the guard would have to model all of them and stay correct as they change.

The priority order settles it. Not changing rendered output and settling on re-run are hard requirements; reflowing wherever possible ranks above converging in one pass, which ranks above never reverting on the render check. A guard here spends the higher goal to buy the lower one. Both hard requirements already hold without it, structurally rather than by prediction: the convergence backstop runs the pipeline to fixpoint and returns the source untouched if it cannot settle, and the render backstop returns the source untouched if the HTML would change. Measured against an unguarded build over every known reproducer, production output was byte-identical to main — same reformatted count, same reverted count, zero unstable — and roughly 19M fuzz executions produced no core-render-changed record from this class, only core-not-idempotent. The residual cost is one extra formatOnce, and Format already loops to fixpoint unconditionally, so no user-visible run is added.

The core idempotency oracle therefore has a documented exception here rather than a bug to root-cause, which is a narrowing of the rule stated under Convergence: divergences are still root-caused by default, and this class is exempt because every candidate fix cost more reflow coverage than the divergence does. Code spans are unaffected either way, having no thematic-break shape to be escaped into.

Segmentation is behind a public interface so it is independently testable and swappable (an NLP segmenter, or something smarter, can be plugged in without touching the pipeline):

// Breaks returns the whitespace gaps separating sentences: for each
// boundary, the [start,end) byte range of the inter-sentence whitespace.
type Segmenter interface {
    Breaks(text string) []Span
}

// Span is a concrete public struct — deliberately not an alias to the
// internal type — so its fields render on pkg.go.dev; Format adapts it
// at the Segmenter boundary (one copy per cluster, custom Segmenters only).
type Span struct{ Start, End int }

Option enums (Mode and Dialect) have exactly one definition, in the internal leaf package internal/opts, which both the root package and the pipeline packages import; the root's exported names are aliases, so the former hand-mirrored copies bridged by unchecked casts (go-quality review S4) cannot exist.

The built-in segmenter ships with a solid default abbreviation list; Options.Abbreviations (and the config file) add to it. Wholesale replacement means providing your own Segmenter.

Whitespace and hard breaks

The formatter, not the segmenter, owns whitespace at boundaries:

  • At a sentence break, the entire inter-sentence whitespace run is consumed and replaced by the newline. mdreflow never emits trailing whitespace — which matters because trailing double-space is Markdown's hard-break syntax.
  • On joins, inter-sentence whitespace normalizes to a single space (required for idempotency; also makes two-spaces-after-period typing invisible).
  • Hard line breaks (trailing double-space, trailing backslash, <br>) carry rendered meaning and are immovable: a hard-break line never joins. Joining is not "concatenate with spaces."
  • The double-space and backslash spellings count as breaks only when goldmark's own inline parse of the document flagged that line ending hard (ast.Text.HardLineBreak, read per line from the paragraph's inline tree). Raw trailing bytes alone cannot tell: an inline extension can consume a line's entire prose and its line ending with it, leaving no text node to carry the flag — the task-list extension does exactly this for a checkbox-only line ("* [X] \n0" renders with a soft break; trusting the raw double-space minted a <br> the renderer never had, #39). This is the "ask goldmark, not a hand grammar" doctrine applied to break detection. A literal <br> is exempt: it is a break wherever it parses as a tag, and its spelling can also be manufactured by a cluster join from bytes no single source line carried — there is no per-line flag to consult for it.

Hard breaks keep the source's own spelling (amendment 2026-08-11). The three spellings are not interchangeable: <br> is raw HTML — goldmark's default configuration strips it, and not every downstream pipeline treats raw HTML the same as Markdown syntax — while a trailing double-space and a trailing backslash are both ordinary, context-sensitive CommonMark syntax. Normalizing every preserved hard break to one configured style meant mdreflow could introduce raw HTML the author never wrote, and fuzzing found four bugs (#40, #47, #49, #52) living in the narrow contexts where the configured spelling didn't actually work at its landing position — each needing a bespoke guard. The policy instead emits the spelling the source used, with one promotion: two trailing spaces are invisible and routinely stripped by editors in transit, so they are promoted to a backslash. Where the promoted backslash cannot work at its landing position — directly after an even, non-zero run of trailing backslashes, where one more would leave three or more (literal text, not a break) — the fallback is two trailing spaces, never <br>: mdreflow never introduces a spelling the source didn't already use.

The promotion is also gated on the dialect (amendment 2026-08-12). Python-Markdown predates CommonMark and never adopted the backslash break: under the mkdocs dialect a trailing backslash renders as a literal backslash followed by an ordinary soft break, so promoting there destroys a working break and prints a stray \ into the page. Only two trailing spaces and a literal <br> carry a break in that renderer, and <br> may never be introduced, so the promotion has no valid target and the source's two spaces are kept. This is the same fallback rule as the escaped-backslash-run position, applied at the level of a whole dialect rather than one line position.

This class of defect is invisible to both safety nets: the render backstop and the fuzz oracle compare through goldmark, where the backslash spelling is a break, so both sides of the comparison agree on output that breaks the published page. Only a real mkdocs build diff can see it, which makes the release procedure's probe page load-bearing rather than a nicety. A source-authored <br> is always kept, canonicalized to lowercase, un-self-closed spelling (<Br /> and <BR> both emit <br>) since a cluster join can manufacture a <br>-shaped tag from bytes no single source line carried, and a canonical spelling is needed for the pipeline to recognize its own output on the next pass. This removes Options.HardBreaks: there is no longer a style to configure. See why-this-is-hard.md for the full context matrix behind this decision.

Typography: removed (2026-08-09)

mdreflow shipped opt-in smart-quote and ellipsis substitution through v0.1.4 and removed it, deliberately. Typography was the only transformation whose purpose was to change rendered output, which made it structurally at odds with everything else here: it forced the render oracle (and now the render backstop) to normalize substitutions back out of both sides — looseness real corruption could hide behind; it was the sole reason for the fuzz harness's one documented scope gate (the caret-zone/typography interaction); it required its own hand-mirrored context scanner to avoid substituting inside HTML attributes and link destinations (issue #3); and it was one of only two byte-rewriters in the pipeline. Removing it, and later removing StripSentenceTerminalBreaks (amendment 2026-08-11: the owner judged it unused and unnecessary), makes the invariant exact: mdreflow never changes what a document renders to. Hard-break spelling changes source bytes, not rendering semantics, and is the one place a comparison of rendered HTML needs a canonicalization rule (the <br> spelling normalization in internal/render) rather than a byte-for-byte match — a comparison-mechanism nuance, not an exception to the invariant itself.

Substitution at render time is the better home for the feature — goldmark's Typographer, Hugo's smartypants, and Python-Markdown's smarty all do it without touching the source — and anyone who wants it baked into source bytes wants a full parse-and-re-emit formatter, which mdreflow is deliberately not. A leftover typography: config key is a loud config error (the YAML parse is strict), never a silent no-op.

The link-reference-definition zone: skip bluntly, by shape

A link reference definition ([label]: /url "title") renders nothing — it is URL metadata — and its grammar is the least reflow-compatible construct in CommonMark: the label, destination, and title may each span onto following lines, a titleless one-line definition can absorb a title from the paragraph after it, and goldmark reorders definition nodes relative to paragraph siblings. A first implementation grew six interlocking adjacency guards (self-completeness reparses, registry-diff title-absorption checks, byte-level titleless backstops, neighbor-spelling symmetry rules) and fuzzing kept finding a seventh shape. The lesson: precision here buys reflow of prose that is rare, ambiguity-laden, and worthless to reflow next to invisible metadata.

The rule is now deliberately blunt and shape-based: any paragraph that contains a non-footnote [label]: shape — original, reflow-escaped (\[label]:), or reflow-joined (text [label]: at line end) spelling — or that sits below such a line within the same contiguous run of non-blank lines, passes through byte-for-byte. Below-in-the-same-run, not merely directly adjacent: a definition's title scan alone can span any number of lines, so a paragraph several lines under the [label]: opener can still be the next text the definition's own scan touches (fuzz-found; a blank line terminates the scan and resets the run). The transitive form is also what makes verdicts stable: everything in a def-containing run is frozen, so the lines a verdict keys on cannot move between passes. No parsing, no adjacency analysis, one predicate plus one precomputed per-line bit. In the common real-world layout (definitions in their own blank-line-separated block) nothing changes: those were already skipped.

One measured narrowing (issue #37): a mid-line shape on a neighbor line counts only when a definition chain can reach it. The blunt rule freezes a paragraph that merely sits under a line whose [label]: shape is mid-line — e.g. a bullet whose sibling above quotes `runnerGroups[0]: …` in a code span (issue #37 measured most of its affected lines in exactly this bracketless-neighbor position). A mid-line shape can only be a live definition when a chain reaches it: definitions are extracted starting at a paragraph's first content and continue back-to-back (a title can close mid-line with the next definition opening immediately after), so a mid-line shape that no chain can reach parses as prose under every rearrangement of the neighbor's bytes. Reachability is judged from line-shape facts alone — the preceding line itself opens def-shaped at its start, or some line above it in the same contiguous run does (the same precomputed per-line bit the transitive rule uses) — never from inline parsing, so the verdict is identical on escaped and unescaped spellings. "Opens def-shaped" tolerates list-marker prefixes: - [a]: /url opens a definition chain inside its item exactly as a bare [a]: /url line does at top level, so marker-prefixed openers arm both the direct neighbor check and the transitive run bit, and the plausible-prefix test that classifies a mid-line shape accepts marker glyphs, digits, and padding while rejecting prose. The narrowing applies only to the neighbor-side checks. The contains check keeps its full anywhere-shape breadth, which preserves both halves of the stability argument: the contains check stays at least as broad as any neighbor check, and the shape-bearing paragraph itself stays frozen, so the line a neighbor's verdict keys on can never move between passes. Boundary-spanning shapes (a label opening on the preceding line and closing on the paragraph's own first line), orphaned ]: closers, and bare caret openers are position-dependent hazards and keep their unnarrowed arms.

Implementation shape: the zone's inputs are computed once per contiguous run as a small per-line facts table (line-start def opener, orphan closer, bare caret opener, mid-line-only shape) instead of ad-hoc scans at each call site, and verdicts are rules over those facts. Same facts, one computation, and each verdict can name the fact that fired — the natural basis for the per-skip reasons proposed in issue #38.

Footnote definitions are exempt and keep reflowing. The caret is a near-perfect discriminator ([^label]: vs [label]: — "near": a caret label that is empty or starts with a space is not a footnote to goldmark but an ordinary definition labeled ^…, and is classified with the definitions, fuzz-found), and the content profile is the opposite: a footnote body is real prose. Two protections replace the guard pile for the caret case:

The neighbor check asks the parser, not the label (amendment 2026-08-14, issue #58). The caret discriminator answers the wrong question for the line above a paragraph. No footnote extension is registered in package gm, so [^1]: /url is an ordinary link reference definition to goldmark, able to take its title from the line below exactly as [docs]: /url is; excluding it by label shape let sentence mode split the paragraph beneath and feed the definition its own first line, deleting that prose from the page. The def-chain-start fact is therefore set by parsing the line in isolation (gm.IsCompleteLinkRefDefLine) as well as by the shape regexes. Asking the parser is what preserves issue #41's coverage rather than reverting it: a real footnote body ([^1]: some ordinary prose) is not a definition to goldmark and answers false, so back-to-back footnote layouts still reflow, while only caret lines that genuinely are definitions ([^1]: /url, [^1]: one) newly freeze the paragraph below. Widening the regexes instead would have frozen every paragraph containing a [^…]: shape — every footnote body in every document.

A paragraph that manufactures a definition falls back alone (amendment 2026-08-14, issue #58). The mirror hazard is not visible in the input at all: a break landing inside a paragraph can strand a label]: fragment at a line start and form a definition that never existed, swallowing the prose around it. There is nothing dangerous to inspect beforehand, so the parser is asked about the candidate output instead — gm.FormsNewLinkRefDef compares the definitions the paragraph's source bytes form against those its reflowed lines form, and any addition means reflow invented one. The fallback is scoped to the one paragraph, which is the point: the document-level render backstop returns src, so a single bad paragraph silently costs the file every other paragraph's reflow. It is verdict-stable by construction — the answer is a function of the paragraph's own source bytes, and a paragraph that falls back emits exactly those bytes, so the next pass re-derives it and settles. The check is deliberately narrow rather than a per-paragraph render comparison: definition formation is decided by the lines the definition occupies, whereas a render diff at this scope would be unsound, since a reference link resolves against definitions elsewhere in the document and a paragraph containing [foo] renders differently alone than in place. For the same reason the window matters: a definition formed from the line above plus the paragraph's first line is invisible inside the paragraph, which is why the neighbor check above is a separate protection rather than one this subsumes.

The exemption applies to a paragraph's own shape only — the zone's neighbor checks have no footnote carve-out (issue #41 removed the one they had). None is needed: a complete footnote-body line ([^1]: body text, the line above in the ordinary back-to-back layout) sets none of the facts the neighbor checks consult, so back-to-back footnotes stay eligible for free — while the caret lines that do set a fact (bare [^label]: openers, orphaned closers) are live definition machinery: like a titleless [label]: line, a bare caret opener completes its destination from the line below it, so joining the paragraph beneath changes whether the definition forms at all. Both spellings were fuzz-found under the old blanket carve-out (" [0]:" and " [^0]:" above a [^0]:0 paragraph), and every line the checks key on is itself frozen, so the uniform rule is verdict-stable.

  • Emission escapes, label-shape-agnostic: an output line that would itself parse as a complete definition (judged empirically — the line is parsed in isolation by goldmark, not matched against a hand-mirrored grammar) or as a bare [label]: opener is backslash-escaped, so reflowed footnote prose can never be swallowed into an accidental definition.
  • Reflowed footnote-body continuation lines are emitted with a 4-space indent. Renderers disagree about whether an unindented lazy-continuation line belongs to the footnote (GitHub's documented convention is to indent); the indented spelling is the one they all keep inside the footnote, and to mdreflow's own parser it is an ordinary paragraph continuation (indented code cannot interrupt a paragraph), so render preservation is unaffected.

One structural exception outranks the exemption: a paragraph whose bytes are double-owned by a sibling definition node passes through. goldmark's duplicate-label handling can extract a repeated definition line as a LinkReferenceDefinition node while also leaving the same bytes as the trailing paragraph's first line (fuzz-found, issue #35: two same-label [^0]: lines then prose). Such a line is live definition machinery whatever its label shape, and reflowing it destroys the neighbor's def shape so the def/paragraph boundary migrates one line per pass — a treadmill no emission escape can stop. The check is AST-only, the same move as the table-adjacency and hidden-line-gap skips: if any sibling definition node's raw lines overlap the paragraph's own byte range, the paragraph is skipped byte-for-byte.

Residual adversarial corners in the caret zone are owned by the emission escapes and the convergence backstop, not by further adjacency guards. (The worst such family — typography substitution verdicts flipping next to [^label]: shapes built out of quote soup — vanished with typography's removal, taking the fuzz harness's one documented scope gate with it.)

No-break spans: ask goldmark, not a hand grammar

(Amendment 2026-08-09, issue #30; supersedes the hand-written inline scanners and the linkify guards built around their blind spots.)

The text whose break points reflow chooses — a cluster's lines joined into one — exists nowhere in the document, so the document parse's AST cannot be queried about it. The original answer was a package of hand-written scanners (segment.NoBreakSpans) approximating CommonMark's inline grammar: code-span pairing, bracket/paren matching for links, a destination-and-title grammar, an inline-HTML tag grammar, an autolink regex. Every one was a mirror that could drift from goldmark, and GFM linkify was the worst: its grammar was deliberately not mirrored, so a family of skip guards existed solely to keep the approximation honest, each costing coverage (#29 measured 2.8 points of sentence-per-line coverage on a real docset).

The replacement parses the joined cluster text through goldmark itself and reads the answer out of the inline AST — the same parser answering the same question the renderer will ask, so drift is structurally impossible. Three decisions make it work:

  1. A paragraph-only block layer. internal/gm builds a second parser configuration sharing the document profile's inline set (including linkify, per dialect) but whose block layer is only goldmark's paragraph parser, with no paragraph transformers. Standalone-parse divergence — joined text starting with - or # parsing as a list or heading, a [label]: dest line vanishing into a definition — cannot happen: every cluster text parses as one Paragraph, which is what it is in the document (spike-confirmed for list/heading/blockquote/fence/definition/thematic-break shapes). Both configurations are constructed from one shared site in internal/gm, so an extension added to the document profile cannot silently miss the span profile.
  2. Spans by complement, not per-construct extents. goldmark's inline Link/Image nodes carry no source extent for their ](dest "title") syntax, so computing each construct's byte range from the AST is a losing game. Inverted, the problem disappears: the breakable regions are the segments of plain Text nodes whose ancestry is only Paragraph/Emphasis/Strikethrough; every other byte of the cluster text — link syntax, code-span interiors and delimiters, autolink URLs, raw HTML — is no-break, computed as the complement. This is conservative by construction: adjacent constructs merge into one protected gap (harmless), and any node kind the walk doesn't recognize defaults to protected.
  3. Reference links need no resolution. The span parse registers no reference definitions, so [text][ref] parses as literal prose rather than the link it may form in the document. Deliberate: the label is the only part of a reference link that lives in the paragraph (its destination lives in the definition, already a skipped zone), and a break inside a label is harmless — label matching collapses internal whitespace, and a soft break inside link text is the same whitespace move reflow makes in any prose. Pre-loading the document's definitions into every cluster parse (parser.Context, spike-confirmed workable) was considered and rejected: machinery for a case that needs none.

One raw-HTML family is excluded from reflow entirely rather than protected as spans: a paragraph containing a <? (processing instruction) or <! (comment/CDATA/declaration) opener outside code spans passes through byte-for-byte (blockmap.hasRawHTMLDeclOpener). These constructs span soft breaks in goldmark's inline grammar, so the walk protects them whole — but a join can land the opener at an output line's start, where HTML blocks of types 2–5 interrupt a paragraph from any line position, forcing emission to backslash-escape it, and the escaped spelling no longer parses as raw HTML: the next pass computes different spans, a parse discontinuity across reflow's own escape (fuzz-found). Inline tags (< + letter) need no such exclusion — their stabilizer (segment.htmlTagOpenerSpans) is a raw-text regex whose verdict is identical for escaped and unescaped spellings. The masking means an opener inside inline code — the common way prose mentions one — costs nothing.

Degenerate parses fall back to no reflow of the cluster. If the parse yields anything but a single Paragraph covering the text (the paragraph parser declines an all-indented line, producing an empty document), the whole cluster is one no-break span and passes through unbroken — coverage lost on a shape that is rare and ambiguous, never correctness.

What stays regex. Constructs invisible to goldmark keep their existing scanners, layered on top as today: inline math ($…$), Hugo shortcodes, MDX {expr}, footnote references (the profile does not enable the footnote extension), and the near-tag opener guard (htmlTagOpenerSpans), which deliberately protects regions goldmark declines to call tags — an idempotency stabilizer the AST by definition cannot provide.

Code-span queries move to the same parse. segment.CodeSpans — used to decide whether a hard-break boundary or cluster join lands inside a genuine code span, and by blockmap's guard-arm masking — shares the linkify blind spot (a backtick inside a bare URL is destination content, not a delimiter, so hand-pairing runs one delimiter out of step). Its answers now come from CodeSpan nodes of the same goldmark configuration (over the paragraph's "\n"-joined lines, matching how goldmark's inline parser sees a paragraph). This, not the no-break rewrite alone, is what retires the linkify guards: with every code-span consumer linkify-aware, hasBacktickInBareURL and the #28 ordering invariant have no blind spot left to cover.

Cost. One paragraph-only parse per cluster measures ~7µs / ~4KB allocated (typical sentence cluster, i7-1165G7) — negligible against the render backstop's two whole-document renders. The spans are computed once per cluster text and shared by every consumer (width canonicalization, the width measurer, word/clause break generation, sentence splitting), which today each recompute them.

Verification. This rewires the heart of span computation, so it ships only behind: a full-corpus differential (fixtures plus the accumulated fuzz corpus) of old spans vs new, expecting byte-identical output everywhere except the reclaimed linkify-guard skips; a long fuzz soak; and the benchmark suite.

Spanning-construct guards: skip only what can actually span

A CommonMark inline link's label and destination can each span a soft line break, so reflow's line-joining and line-splitting can change what they parse to. A preparation step and three whole-paragraph skip arms defend this, each scoped to the shape that can actually start the hazard. (These arms guard a block-level hazard — reflow's line rearrangement letting a link reference definition consume prose — which no per-cluster span computation addresses; they are untouched by the ask-goldmark rewrite above, except that their masking step now shares its parse.)

Code-span masking. The three arms scan a copy of the paragraph with every closed inline code span's interior replaced by filler bytes (same byte geometry, spans read from the ask-goldmark parse above). A bracket or paren inside a code span is literal and can open nothing, so it arms no guard — paragraphs that merely document Markdown or YAML syntax (`runs-on: [self-hosted,` wrapping across a line) reflow. An unmatched backtick run opens no span and masks nothing, so a bracket after one still arms normally.

  • Bracket arm: a [ left structurally open at a line's end (even one a later line closes — the span across the break is itself the hazard), but only in a paragraph that also contains a ]: — on one line, or as ] ending one line with : opening the next — the only shape a link reference definition can be built from. Without it, no rearrangement of the lines can form a definition (CommonMark permits a newline inside every other bracket construct's label or text, and label matching collapses internal whitespace), so a wrapped link is safe to reflow — and needs to be, since a skipped-because-wrapped paragraph is otherwise a permanent fixed point: the skip preserves the very break that causes it.
  • Destination-paren arm: a ( left open at a non-final line's end skips the paragraph only if that specific paren was opened by ]( — the only spelling that opens an inline destination, since CommonMark admits no whitespace between a link's ] and its (. Each open paren carries its own armed flag (a stack, not a depth-0 check: the hazard nests inside ordinary prose parens). An ordinary prose parenthetical spanning a line break arms nothing and keeps reflowing regardless of what brackets appear elsewhere in the paragraph.
  • Angle-destination arm: ]( then optional spaces then < with no > on the same line — joining can complete it into a real angle-bracket destination.

Definition titles in parens (which may also span lines) are caught upstream by the link-reference-definition zone, not by these arms.

Control-character paragraphs pass through

A paragraph containing a C0 control byte other than tab (form feed, vertical tab, a bare carriage return outside a CRLF pair, ...) passes through byte-for-byte. No text editor produces these inside prose; they exist in fuzz inputs, where they sit in exactly the grammar corners (indentation width, whitespace-class membership) that differ between parser implementations. Reflowing around them buys nothing for real documents and costs a long tail of corner-case hardening. CRLF line endings are unaffected (the \r of a CRLF pair is line-ending machinery, not paragraph interior).

Guarantees

Stated as testable promises, enforced by the harness in Testing.

Input domain: valid UTF-8. Format rejects invalid UTF-8 with the typed error ErrInvalidUTF8 and writes nothing; the guarantees below are stated over inputs it accepts. (The CLI already refused such files via its binary sniff, but the sniff only reads the first 8 KB — the library check is total. Markdown is text; reflowing bytes that have no character interpretation was never meaningful, and fuzzing spent most of its findings on inputs no user could produce with a text editor.)

  1. Idempotency. Format(Format(x)) == Format(x) for every accepted input and option set. This is structural, not aspirational: Format runs to fixpoint and falls back to the original text for anything that will not converge (see Convergence).
  2. Render preservation. Reflow does not change the rendered document, verified by comparing goldmark HTML output before and after. Hard-break spelling changes source bytes, not rendering semantics (see "Whitespace and hard breaks"); no other exception exists.
  3. Byte-identical pass-through. Everything outside reflowed paragraph prose is emitted byte-for-byte.
  4. Check mode. --check and --diff report unformatted files without writing, with a stable exit-code contract for CI and agents.

Library API

Library-first: the root package is the product, cmd/mdreflow is a thin wrapper, internals live in internal/. The core is pure bytes-in/bytes-out; file discovery, config loading, excludes, and input detection are CLI-layer concerns (promoting them into the library later is non-breaking; the reverse is not).

package mdreflow // import "github.com/jbeda/mdreflow"

type Mode int
const (
    ModeSentence Mode = iota // default
    ModePara
    ModeWrap
)

type Options struct {
    Mode          Mode
    MaxWidth      int       // 0 = unbounded; otherwise >= MinMaxWidth (20); on ModeSentence enables secondary clause breaks
    Abbreviations []string  // additions to the built-in list
    Segmenter     Segmenter // nil = built-in
}

func Format(src []byte, opts Options) ([]byte, error)
func NeedsFormat(src []byte, opts Options) (bool, error)

// Invalid UTF-8 input; returned (wrapped) by Format/Check/FormatReader.
// Callers branch with errors.Is.
var ErrInvalidUTF8 = errors.New("input is not valid UTF-8")

// Convenience wrapper; buffers the full input (Markdown is not streamable —
// e.g. a reference definition on the last line affects the first).
func FormatReader(dst io.Writer, src io.Reader, opts Options) error

func DefaultAbbreviations() []string

Every zero value is the default (Options{} is valid and sensible): zero Mode is sentence mode, zero MaxWidth is unbounded.

The width floor (amendment 2026-08-09): a non-zero MaxWidth below MinMaxWidth (20) is an options error, in the library and the CLI alike. Nearly every pathological width finding from the fuzz campaign needed single-digit widths, where geometry forces breaks inside constructs; no human asks for width 12. Refusing the range deletes that adversarial surface from the product without weakening the harness: like the convergence backstop, the floor has a test-only disable so the fuzzer and the narrow-width fixtures keep driving the unrestricted core, and a tiny-width find is still root-caused (some generalize to legal widths with long tokens) — it just can no longer be a user-facing bug on its own. No constructor needed.

[]byte (not string, not streams) because goldmark's API and AST offsets are byte-based, os.ReadFile hands you bytes, and a string API would force copies both directions for no gain.

CLI

mdreflow [flags] [path ...]
  • Paths: files and/or directories; directories are walked recursively for Markdown extensions (.md, .mdx, .markdown). Formatting is in-place when paths are given — this is a batch tool for pre-commit hooks and agents, where in-place is what you mean.
  • stdin/stdout: no paths (or -) reads stdin and writes stdout. This is the pipe mode, and it gives any editor with a "filter through command" binding format-on-demand without an extension.
  • Flags: --mode, --max-width, --check, --diff, --stdout, --force, --config, --no-gitignore, --explain. Stdlib flag; the surface is one command.
  • --explain (#38): every skip site in blockmap.build (and collect's depth cap) reports a distinct reason; the flag prints, per frozen paragraph, its line range, a stable machine-legible reason code, and a remediation hint drawn from why-this-is-hard.md's "What authors can do" section (never backslash-escaping — the zone is escape-tolerant on purpose). Output goes to stderr so stdout stays clean in pipe and --diff modes; diagnostics only, output bytes and exit codes unchanged. The same report is the library's Explain; the reason a paragraph froze is diagnosable by agents without hand-classifying diffs.
  • Exit codes (a contract, since agents branch on them): 0 success/clean; 1 would-reformat (--check/--diff only); 2 usage or config error; 3 refused input (not Markdown, or excluded without --force).
  • --help is a first-class deliverable: complete flag docs, the exit-code contract, and examples in the help text itself, so an unattended agent can self-serve without a README.

Configuration

.mdreflow.yaml, discovered upward from each target file. Defaults should be good enough that most repos need no file. Precedence: flags > config file > built-in defaults.

mode: sentence        # sentence | para | wrap
max-width: 0
abbreviations:        # additions to the built-in list
  - "et al."
exclude:              # gitignore syntax
  - "CHANGELOG.md"
  - "generated/**"

Excludes

  • The repo's .gitignore is respected by default (--no-gitignore disables) — node_modules and build output are almost always already covered.
  • The config exclude: list (gitignore syntax) covers tracked-but-generated files.
  • Built-in always-excludes underneath: .git/, node_modules/, vendor/.
  • Excludes apply even to explicitly named files — the flowmark #43 failure, designed out. A skipped explicit path prints skipped (excluded by …); --force overrides.

Candidate libraries (implementation-time decision; the behavior is the commitment, the library is a detail): boyter/gocodewalker for tree walking that honors nested .gitignore, and sabhiram/go-gitignore for matching config patterns. Caveat: no Go gitignore implementation is fully spec-compliant (see go-git#108), so the test suite compares our matching against git check-ignore on a synthetic tree.

Non-Markdown input detection

The observed failure mode is an unattended agent running the formatter on a YAML or JSON file. V1 defense, deliberately cheap:

  • Extension check on explicitly named files: known-Markdown extensions pass; known-other extensions (.yaml, .json, .go, …) are refused.
  • Binary sniff: NUL bytes or invalid UTF-8 in the first 8 KB → refused.

Refusal is exit code 3 with a one-line explanation and no write; --force overrides. Refusing (vs. warning) is the point: a warning in scrolled-past output doesn't stop an unattended agent, a non-zero exit does. Directory walks are unaffected (they only pick up Markdown extensions). Content-based sniffing for unknown extensions ("parses as YAML/JSON with no Markdown block constructs") is deferred until mislabeled files are observed in practice — the heuristic's second half is fuzzy, and the extension check covers the failures actually seen.

Testing

The wrapping logic is heuristic; a deep test corpus is the only durable defense against regressions, and the guarantees above are only as real as this harness.

  1. Golden-file fixtures (testdata/): input → expected output pairs covering every skip-list construct, every mode, every option. Fixture content is synthetic (lorem-ipsum-style or purpose-written), reproducing structures observed in real repos without lifting content.
  2. Property tests over the corpus: idempotency and render preservation (goldmark HTML before == after, documented exceptions aside) run automatically over every fixture — adding a fixture adds the property checks for free.
  3. Mined regression cases: bugs from flowmark's, mdslw's, and rumdl's issue trackers re-authored (not copied) as fixtures — e.g. flowmark #68's `token`. boundary misses.
  4. Segmenter suite: the sentence splitter tested standalone against a Golden-Rules-style case list (in the style of pragmatic_segmenter's golden rules, cases our own).
  5. Exclude parity: our gitignore matching vs. git check-ignore output on a synthetic tree.
  6. Fuzzing: Go native fuzzing on Format with crash, idempotency, and render preservation as oracles. Invalid-UTF-8 inputs assert only the ErrInvalidUTF8 refusal (that path must still never panic); the reflow oracles run on accepted inputs. The harness has no scope gates: every accepted input runs every oracle. (The one historical gate — caret-def openers mixed with typography-substitutable quotes — existed only for typography and was deleted with it; see the Typography section.)

Dependencies

Small on purpose; additions require amending this doc.

  • github.com/yuin/goldmark (+ GFM extension) — parsing, library core
  • github.com/goccy/go-yaml — config file, CLI layer only. (Amended 2026-08-07: originally gopkg.in/yaml.v3, which is no longer maintained; goccy/go-yaml is actively maintained with better spec compliance.)
  • YAML front matter is recognized by our own byte-range pre-scan (internal/blockmap), not a parser extension. (Amended 2026-08-07: this was goldmark-meta at first, but mdreflow never consumes the parsed metadata — it only needs the block excluded from reflow — and goldmark-meta's YAML parsing dragged in unmaintained yaml.v2 plus parser-hook artifacts the fuzz harness had to defend against. Opener is exactly --- at byte 0; unterminated front matter is not front matter.)
  • github.com/boyter/gocodewalker — CLI layer only; used solely for its vendored go-gitignore subpackage (nested-.gitignore semantics). The walk itself is hand-rolled filepath.WalkDir (M4 decision; sabhiram/go-gitignore was evaluated and dropped — no nested-file support).
  • github.com/pmezard/go-difflib — CLI layer only, unified diffs for --diff (M4 addition; hand-rolling Myers diff wasn't worth the risk).
  • Stdlib flag for the CLI (single command; cobra buys weight, not function; the cost is shell completions, acceptable for now)

Milestones

Vertical slice first — a thin end-to-end path that is immediately usable and carries the test harness from day one — then broaden.

  • M0 — spike: goldmark vs. MDX/Docusaurus/Hugo input. Where do JSX blocks, {} expressions, ::: directives, and shortcodes land in the AST? Decides whether a fence-scanning pre-pass is needed. This is the load-bearing unknown; it runs before any pipeline code hardens.
  • M1 — vertical slice: mdreflow < in.md > out.md. Sentence mode only; pipeline (parse → paragraph map → join → segment → emit); minimal skip-list (code fences, front matter); built-in segmenter with default abbreviations; golden harness with idempotency + render-preservation properties. Usable via pipes and editor filters from this point on.
  • M2 — correctness breadth: full skip-list, no-break inline spans, hard-break styles and whitespace rules, list/blockquote continuation indentation, mined regression fixtures, fuzzing.
  • M3 — modes: para, wrap, --max-width secondary clause breaks.
  • M4 — CLI proper: path/directory handling, in-place writes, config discovery, excludes + gitignore, --check/--diff, input detection, exit codes, agent-grade --help.
  • M5 — ship: typography flags, goreleaser binaries, .pre-commit-hooks.yaml, README and docs polish.

Versioning: v0.x until the library API has survived real use; v1.0.0 is an API-stability promise (Go module semantics make it binding). CI: GitHub Actions — test, vet, golangci-lint on push; goreleaser on tags.

Future directions

Recorded so the design doesn't preclude them; none are commitments.

  • Markdown embedded in YAML/JSON values — the original wishlist item, now judged a large scope expansion of uncertain value. The hard part is knowing which fields are Markdown; the shape would be caller-supplied yq/jq-style path selectors plus comment-preserving YAML node surgery, with mdreflow formatting the extracted strings. The bytes-in/bytes-out library API is the only accommodation made now: it keeps this buildable from outside.
  • Per-file options in front matter — a reserved key (e.g. mdreflow: {mode: para}) slotting into the precedence chain as flags > front matter > config > defaults.
  • --dialect flag — subsets of the tagged skip-list rules.
  • Content sniffing for unknown extensions — gated on observed need.
  • GitHub Action, Homebrew tap, editor extensions — distribution beyond goreleaser + pre-commit.
  • Alternative segmenters — NLP (e.g. jdkato/prose) or model-based splitting behind the Segmenter interface, if the regex approach's ceiling is ever actually hit.