README
¶
mdreflow
Reflow Markdown prose. The default mode is sentence-per-line (semantic line breaks); paragraph-per-line and classic hard wrap share the same pipeline. It is a Go library first, with a thin CLI on top.
mdreflow changes where lines break inside paragraph prose and touches nothing else. It never rewrites block structure (headings, list markers, tables, escaping) and produces output by splicing reflowed prose into the original bytes, so everything outside a reflowed paragraph passes through byte-for-byte. If you also want lint-style normalization, pair it with a tool like rumdl; the two touch disjoint parts of the file.
Why sentence-per-line? Diffs. One changed sentence is one changed line, which makes prose reviews readable and gives both humans and agents a stable convention for edits.
Status: early v0. The library API and CLI surface are still moving; every milestone in the design doc is implemented, but nothing is frozen until v1.0.0.
Install
Via Homebrew:
brew install jbeda/tap/mdreflow # macOS
brew install --cask jbeda/tap/mdreflow # Linux (Homebrew 4.5+, preliminary cask support)
Prebuilt binaries for Linux, macOS, and Windows (amd64 and arm64) are on the releases page, with a checksums.txt alongside.
Unpack the archive and put mdreflow on your PATH.
Or build from source:
go install github.com/jbeda/mdreflow/cmd/mdreflow@latest
Usage
mdreflow --help is the complete reference and stays canonical: every flag, the exit-code contract, the config format, and worked examples.
The sections below cover the parts you will look up most.
mdreflow docs/ # format a tree in place (respects .gitignore + excludes)
mdreflow --check docs/ # CI gate: exit 1 if anything would change
mdreflow --diff README.md # show what would change
mdreflow --explain docs/ # report each frozen paragraph and why, to stderr
mdreflow < in.md > out.md # pipe mode, e.g. an editor filter binding
With path arguments, mdreflow formats in place: files directly, directories by walking them for .md, .mdx, and .markdown files.
With no arguments (or -), it reads stdin and writes stdout; --check and --diff work there too, reporting the input as -.
Never redirect a file onto itself in pipe mode (the shell truncates it first); use in-place mode for that.
Some paragraphs are deliberately never reflowed — constructs like link-reference-definition shapes make reflow unsafe, so mdreflow preserves them byte-for-byte (why).
--explain reports each one to stderr with its location, a stable machine-legible reason code, and a remediation hint; it combines with every mode and never changes output or exit codes.
The same report is available as a library call, mdreflow.Explain.
As a library:
import "github.com/jbeda/mdreflow"
out, err := mdreflow.Format(src, mdreflow.Options{})
Options{} is valid and sensible: sentence mode, no width limit.
The full API reference is on pkg.go.dev.
Modes
sentence(default): one sentence per source line.--max-widthoptionally adds clause-level breaks inside sentences that run past the limit; the default of 0 means unbounded. Non-zero widths below 20 are refused in every mode — very narrow wrapping forces breaks inside Markdown constructs.para: each paragraph joined onto a single line.--max-widthis an error here.wrap: classic hard wrap at--max-width(default 80).
Dialects
The default dialect, gfm, is the permissive GitHub-flavored superset: GFM extensions plus footnotes.
Docusaurus, Hugo, and MDX constructs are recognized and passed through untouched in every dialect.
--dialect mkdocs additionally reflows MkDocs/Python-Markdown admonition bodies (!!! note followed by a 4-space-indented body), which CommonMark parsers can only see as indented code blocks — opt-in because reflowing one changes what a CommonMark renderer emits.
Recognition is deliberately narrow: bodies containing a fence marker or more than one paragraph, indented past 4 spaces, or whose wrap would need a backslash escape are left alone.
The dialect also decides how a hard line break is written. mdreflow keeps the spelling the source used and promotes a trailing double-space to a backslash, since double spaces are invisible and get stripped in transit — but not under mkdocs, where Python-Markdown has no backslash hard break and renders one as a literal \.
There, a double-space is kept as written.
Raw HTML (<br>) is never introduced in any dialect.
Configuration
mdreflow looks for .mdreflow.yaml by walking upward from each target file (from the current directory in pipe mode), or reads the file given with --config.
Precedence is flags > config file > built-in defaults, and a flag explicitly given on the command line wins even if its value equals the default.
Unknown keys and unrecognized values are a loud error (exit 2), not a silent no-op, so a typo cannot quietly change behavior.
mode: sentence # sentence | para | wrap
dialect: gfm # gfm | mkdocs (mkdocs also reflows admonition bodies)
max-width: 0
abbreviations: # additions to the built-in list
- "et al."
exclude: # gitignore syntax, matched like a .gitignore
- "CHANGELOG.md"
- "generated/**"
These are all the keys; --explain is flag-only.
Excludes
Excludes are checked in order, first match wins: the built-in always-excludes (.git/, node_modules/, vendor/), then the repository's .gitignore files (nested ones included; --no-gitignore disables), then the config file's exclude: patterns.
A directory walk skips excluded files silently.
A file named explicitly on the command line is refused loudly instead (exit 3), so an agent told to format a generated file finds out rather than silently succeeding; --force overrides.
Exit codes
0: nothing needed to change.1:--checkor--difffound at least one file that would be reformatted; nothing was written.2: usage or config error; the run aborts immediately.3: at least one input was refused (excluded, or not recognized as Markdown) without--force; the rest of the batch still runs.
When several apply in one run, the most severe wins: 2 > 3 > 1 > 0. The numbers follow Unix convention (diff-style 1, usage-error 2), so severity does not track numeric order.
pre-commit
repos:
- repo: https://github.com/jbeda/mdreflow
rev: v0.3.0 # or any commit
hooks:
- id: mdreflow # formats staged Markdown in place
# - id: mdreflow-check # or: fail the commit instead of rewriting
Both hooks cover .md, .mdx, and .markdown.
The mdreflow hook rewrites files and lets pre-commit re-stage them.
The mdreflow-check hook writes nothing and fails if anything would change, which is the shape you want in CI.
Documentation
- pkg.go.dev/github.com/jbeda/mdreflow is the library API reference, rendered from the doc comments.
- docs/design.md is the canonical design: goals, modes, architecture, dialect handling (GFM, MDX/Docusaurus, Hugo), guarantees, API, CLI, and milestones. Design changes land there before code.
- docs/why-this-is-hard.md explains why reflowing Markdown safely is hard — the block/span/cluster model, the ambiguities that force the freeze-based design, and what authors can do when a paragraph will not reflow.
- docs/m0-spike-findings.md maps how dialect constructs land in goldmark's AST and why the skip-list works the way it does.
License
Apache-2.0.
Documentation
¶
Overview ¶
Package mdreflow reflows Markdown prose: it changes where lines break inside paragraph text and leaves everything else in a document untouched. Its home mode is sentence-per-line (one sentence per source line, aka semantic line breaks); paragraph-per-line and hard-wrap modes share the same pipeline (see docs/design.md).
mdreflow is a reflow tool, not a formatter: it never rewrites block structure (heading style, list markers, tables, escaping). The Markdown parser is used read-only to locate paragraph prose; output is produced by splicing reflowed prose into the verbatim source bytes.
The library API in docs/design.md is complete: all three modes, MaxWidth, hard-break spelling, the abbreviation list, and a pluggable Segmenter are all implemented.
Index ¶
- Constants
- Variables
- func DefaultAbbreviations() []string
- func Format(src []byte, opts Options) ([]byte, error)
- func FormatReader(dst io.Writer, src io.Reader, opts Options) error
- func NeedsFormat(src []byte, opts Options) (bool, error)
- type Dialect
- type FrozenParagraph
- type Mode
- type Options
- type Segmenter
- type SkipReason
- type Span
Constants ¶
const MinMaxWidth = 20
MinMaxWidth is the smallest non-zero Options.MaxWidth Format accepts — see Options.MaxWidth's doc comment for why tiny widths are refused.
Variables ¶
var ErrInvalidUTF8 = errors.New("mdreflow: input is not valid UTF-8")
ErrInvalidUTF8 is returned (possibly wrapped) by Format, Check, and FormatReader when the input is not valid UTF-8. Markdown is text; bytes with no character interpretation have nothing meaningful to reflow. Callers branch with errors.Is.
Functions ¶
func DefaultAbbreviations ¶
func DefaultAbbreviations() []string
DefaultAbbreviations returns a copy of the built-in sentence segmenter's abbreviation exception list (see docs/design.md's "Sentence segmentation" section). Options.Abbreviations adds to this list; it is never replaced wholesale short of providing a custom Segmenter.
func Format ¶
Format reflows src according to opts and returns the result. Everything outside reflowed paragraph prose — code blocks, front matter, headings, lists, blockquotes, tables, and so on — is returned byte-for-byte (see docs/design.md's "Guarantees" section).
Format returns an error, without partial output, if src is not valid UTF-8 (ErrInvalidUTF8) or opts is invalid: see the doc comments on Mode and Options.MaxWidth.
Format's output is a fixpoint: formatting it again returns it unchanged. In the vanishingly rare case where reflow will not converge (docs/design.md, Convergence), Format returns src as-is rather than unstable output.
func FormatReader ¶
FormatReader reads all of src, formats it per opts, and writes the result to dst. Markdown is not streamable in general — e.g. a reference link definition on the last line can affect how the first paragraph renders — so FormatReader buffers the full input before writing anything.
func NeedsFormat ¶ added in v0.1.5
NeedsFormat reports whether Format would change src, without writing anything: true means src is not in formatted form. (Renamed from Check pre-v1: "ok, err := Check(...)" — the shape a reader expects from that name — read exactly backwards.)
Types ¶
type Dialect ¶ added in v0.1.5
Dialect names the renderer profile a document tree targets. It is a single-select enum like Mode, and a bundle rather than a feature flag: a dialect selects which flavor-specific block recognitions are on (docs/design.md, "Dialects: renderer profiles, and the skip-list").
const ( // DialectGFM is the default: the permissive GitHub-flavored superset // (GFM extensions plus footnotes). There is deliberately no // "commonmark" value — that name is kept for a possible future // strict profile with the GFM extensions off, and aliasing it to the // default would burn the name on the one thing it doesn't accurately // describe. DialectGFM Dialect = opts.DialectGFM // DialectMkDocs additionally reflows MkDocs / Python-Markdown // admonition bodies ("!!! note" plus a 4-space-indented body). A // CommonMark parser can only see such a body as an indented code // block, so reflowing it changes what a CommonMark renderer emits — // which is why it is opt-in and can never be the default. DialectMkDocs Dialect = opts.DialectMkDocs )
type FrozenParagraph ¶ added in v0.1.7
type FrozenParagraph struct {
StartLine, EndLine int
Reason SkipReason
// Detail is a short human-readable statement of what fired, phrased
// to follow "skipped: ".
Detail string
// Remediation says how to make the paragraph reflowable, when there
// is something to do; frozen paragraphs are always preserved
// byte-for-byte, so hand-formatting once is always an option.
Remediation string
}
FrozenParagraph reports one paragraph Format passes through byte-for-byte instead of reflowing: where it is, which guard froze it, and what an author can do about it. Line numbers are 1-based and inclusive, counted in the source given to Explain.
func Explain ¶ added in v0.1.7
func Explain(src []byte, opts Options) ([]FrozenParagraph, error)
Explain reports every paragraph Format would pass through byte-for-byte instead of reflowing, in source order, with the guard that froze it and a remediation hint. It is purely diagnostic: it never writes anything, and Format's output is unaffected by whether Explain runs.
Explain judges src as given (the same verdicts Format's first pass uses). Options participate the same way they do in Format — the dialect selects block recognition — and invalid options or non-UTF-8 input fail with the same errors Format returns.
type Mode ¶
Mode selects the top-level reflow strategy.
(Mode and its constants are aliases of one shared definition in an internal leaf package the pipeline uses directly, so the public and internal values can never drift; this alias is the documented, supported name.)
const ( // ModeSentence joins each paragraph's lines and splits at sentence // boundaries, one sentence per line. The default mode. ModeSentence Mode = opts.ModeSentence // ModePara joins each paragraph (each hard-break cluster) to a single // line, with no further splitting. // Options.MaxWidth must be 0 in this mode: para mode's whole point is // one unconditional line, so a non-zero MaxWidth has nothing to apply // to and Format returns an error rather than silently ignoring it — // see Options.MaxWidth's doc comment. ModePara Mode = opts.ModePara // ModeWrap hard-wraps at Options.MaxWidth, breaking at word // boundaries only: a no-break span or a single word wider than the // limit overflows rather than being split. Options.MaxWidth of 0 // defaults to 80 in this mode only — see Options.MaxWidth's doc // comment. ModeWrap Mode = opts.ModeWrap )
type Options ¶
type Options struct {
// Mode selects the reflow strategy. Zero value is ModeSentence.
Mode Mode
// Dialect opts into flavor-specific block recognition. Zero value
// is DialectGFM (today's behavior); see the Dialect constants.
Dialect Dialect
// MaxWidth bounds line length, measured in runes — not bytes, and not
// Unicode grapheme clusters or East-Asian display width. That is a
// deliberate, pragmatic simplification (the same starting point every
// shipping line-wrap tool uses); full grapheme-cluster/width-aware
// measurement is a documented future refinement, not implemented
// here. In every mode, a no-break span (inline code, a link, ...) or
// a single word wider than the limit overflows rather than being
// split.
//
// A non-zero MaxWidth below MinMaxWidth (20) is an options error.
// Nearly every pathological width behavior the fuzz campaign found
// needed single-digit widths, where the geometry forces breaks
// inside constructs; no real document wants width 12. Refusing the
// range outright deletes that adversarial surface from the product
// (docs/design.md, "The width floor").
//
// MaxWidth's meaning is mode-dependent:
//
// - ModeSentence: 0 (default) leaves long sentences alone. A
// non-zero value adds a secondary break to any sentence longer
// than MaxWidth, at the last clause boundary (a comma or
// semicolon followed by a space) before the limit, falling back
// to the last word boundary before the limit, falling back to an
// overflowing single word or no-break span if neither exists.
// - ModeWrap: classic hard wrap at word boundaries. 0 means 80 in
// this mode specifically, matching the CLI's --max-width default
// (docs/design.md's CLI table) — the one place in this package a
// zero MaxWidth does not mean "unbounded".
// - ModePara: MaxWidth must be 0. A paragraph always joins to a
// single line unconditionally in this mode, so Format returns an
// error for a non-zero MaxWidth rather than silently ignoring it.
MaxWidth int
// Abbreviations adds to (never replaces) the built-in segmenter's
// abbreviation exception list. Ignored if Segmenter is set.
Abbreviations []string
// Segmenter overrides the built-in sentence segmenter. Nil (default)
// uses the built-in regex/abbreviation-list segmenter seeded with
// DefaultAbbreviations plus Abbreviations.
Segmenter Segmenter
}
Options configures Format, NeedsFormat, and FormatReader. The zero value is the default and is always valid: sentence mode, unbounded width, the GFM dialect, the built-in segmenter with its default abbreviation list.
type Segmenter ¶
type Segmenter interface {
// Breaks returns the whitespace gaps separating sentences in text: for
// each boundary, the [start,end) byte range of the inter-sentence
// whitespace. The caller (not the Segmenter) decides what replaces
// each gap.
Breaks(text string) []Span
}
Segmenter finds sentence boundaries in prose text. It is independently testable and swappable: provide Options.Segmenter to plug in something smarter than the built-in regex/abbreviation-list segmenter.
type SkipReason ¶ added in v0.1.7
type SkipReason string
SkipReason is a stable, machine-legible code naming the guard that froze a paragraph. Codes are part of the public surface: agents and scripts may branch on them. New codes may be added; existing ones do not change meaning.
const ( // SkipLinkRefDefShape: the paragraph contains a line shaped like a // link reference definition ("[label]: ..."). SkipLinkRefDefShape SkipReason = "link-ref-def-shape" // SkipLinkRefDefNeighbor: the paragraph sits directly against link- // reference-definition machinery above it, within a definition's // possible reach. SkipLinkRefDefNeighbor SkipReason = "link-ref-def-neighbor" // SkipPossibleLinkRefDef: reflow could complete an unbalanced "[" or // an unclosed destination into a real definition. SkipPossibleLinkRefDef SkipReason = "possible-link-ref-def" // SkipRawHTMLDeclOpener: a raw "<?" or "<!" opener outside a code // span. SkipRawHTMLDeclOpener SkipReason = "raw-html-decl-opener" // SkipUnterminatedTag: the paragraph opens with an HTML/JSX tag whose // closing ">" is on a later line. SkipUnterminatedTag SkipReason = "unterminated-tag" // SkipTableAdjacency: the paragraph sits directly under a table with // no blank line between them. SkipTableAdjacency SkipReason = "table-adjacency" // SkipDeepNesting: the paragraph is nested more than two container // (list/blockquote) levels deep. SkipDeepNesting SkipReason = "deep-nesting" // SkipControlBytes: the paragraph contains a C0 control byte other // than tab, CR, or LF. SkipControlBytes SkipReason = "control-bytes" // SkipDegenerateBlank: the paragraph (or one of its lines) trims to // nothing — a control-character parser artifact. SkipDegenerateBlank SkipReason = "degenerate-blank" // SkipHiddenLineGap: the parser carved another construct's bytes out // of this paragraph's line range. SkipHiddenLineGap SkipReason = "hidden-line-gap" // SkipDoubleOwnedLine: a line belongs to both this paragraph and a // duplicate link-reference definition. SkipDoubleOwnedLine SkipReason = "duplicate-def-line" // SkipDialectBlock: the block is a recognized non-prose dialect // construct (front-matter fence, math block, MDX, shortcode). SkipDialectBlock SkipReason = "dialect-block" )
type Span ¶
type Span struct {
Start, End int
}
Span is a half-open byte range [Start, End) into the text passed to a Segmenter's Breaks method. It is a concrete struct (not an alias to an internal type) so its fields are visible in rendered documentation; Format adapts it at the Segmenter boundary.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
mdreflow
command
Command mdreflow reflows Markdown prose: sentence-per-line by default, paragraph-per-line or hard-wrap on request.
|
Command mdreflow reflows Markdown prose: sentence-per-line by default, paragraph-per-line or hard-wrap on request. |
|
internal
|
|
|
blockmap
Package blockmap derives the set of reflow-eligible paragraphs from a goldmark AST, anywhere in the document: top-level, and nested inside any depth of List/ListItem/Blockquote nesting.
|
Package blockmap derives the set of reflow-eligible paragraphs from a goldmark AST, anywhere in the document: top-level, and nested inside any depth of List/ListItem/Blockquote nesting. |
|
config
Package config loads and discovers .mdreflow.yaml configuration files (docs/design.md's "Configuration" section) and merges them with CLI flag values under the documented precedence: flags > config file > built-in defaults.
|
Package config loads and discovers .mdreflow.yaml configuration files (docs/design.md's "Configuration" section) and merges them with CLI flag values under the documented precedence: flags > config file > built-in defaults. |
|
exclude
Package exclude decides whether a path should be skipped by mdreflow's CLI: repo .gitignore patterns, config-file exclude patterns, and a small set of always-excluded directories (docs/design.md's "Excludes" section).
|
Package exclude decides whether a path should be skipped by mdreflow's CLI: repo .gitignore patterns, config-file exclude patterns, and a small set of always-excluded directories (docs/design.md's "Excludes" section). |
|
gm
Package gm provides the single goldmark configuration used throughout mdreflow: by the reflow pipeline to parse source Markdown, and by tests to render HTML for the render-preservation property check.
|
Package gm provides the single goldmark configuration used throughout mdreflow: by the reflow pipeline to parse source Markdown, and by tests to render HTML for the render-preservation property check. |
|
opts
Package opts holds the option types shared between the public mdreflow package and the internal pipeline packages.
|
Package opts holds the option types shared between the public mdreflow package and the internal pipeline packages. |
|
reflow
Package reflow implements mdreflow's reflow pipeline: given a parsed goldmark document and its source bytes, join each reflow-eligible paragraph's prose lines, find sentence boundaries, and splice the result back into the verbatim source.
|
Package reflow implements mdreflow's reflow pipeline: given a parsed goldmark document and its source bytes, join each reflow-eligible paragraph's prose lines, find sentence boundaries, and splice the result back into the verbatim source. |
|
render
Package render renders Markdown through the same goldmark configuration the reflow pipeline parses with (package gm) and normalizes the HTML for render-preservation comparison.
|
Package render renders Markdown through the same goldmark configuration the reflow pipeline parses with (package gm) and normalizes the HTML for render-preservation comparison. |
|
segment
Package segment implements mdreflow's built-in sentence segmenter and the no-break inline span detector the reflow pipeline uses to protect inline code and links from being split.
|
Package segment implements mdreflow's built-in sentence segmenter and the no-break inline span detector the reflow pipeline uses to protect inline code and links from being split. |