Skip to content

OTL/xort

Repository files navigation

xort

CI codecov crates.io docs.rs License: MIT

xort sorting 1M integers ~4x faster than GNU sort

A fast, modern, parallel drop-in replacement for the Unix sort command.

xort aims to be to sort what ripgrep is to grep and fd is to find: compatible enough to drop into your existing scripts and muscle memory, but parallel by default, measurably faster, and with a few things the classic tool simply can't do.

Status: feature-complete for v1. GNU-compatible sort with field keys and all type comparators, external merge sort, fused --top/--count, CSV/JSON awareness, and UX polish — differentially tested against GNU sort (155 cases).

Contents

Features

  • Parallel by default — uses every core. ~1.7–2.4× faster than GNU sort on text, and an integer radix fast path makes -n ~2.5× faster on whole numbers (and never slower elsewhere — it transparently falls back).
  • 🎯 Drop-in compatible — the GNU flags you already know (-n -k -t -u -r -s -c -m -h -V -M -g -b -f -z -o -S -T --parallel), with output byte-identical to LC_ALL=C sort, differentially tested against GNU (155 cases).
  • 🧠 Every ordering GNU has — numeric, general-float, human sizes (2K/1G), version/natural (v2 < v10), and month — plus date/time (--date-sort: ISO-8601, Unix epoch, Apache/syslog logs).
  • 📊 Things GNU can't do--top N (bounded selection, far cheaper than sort | head), --count (built-in sort | uniq -c), and --header (pin the header row, sort the rest).
  • 🗂️ Structured data — quoting-correct --csv/--tsv (sort by column name or index) and --json/--jsonl (sort by a .path.to.field).
  • 🗜️ Transparent compression — gzip/zstd input auto-detected by magic bytes (works for stdin and mis-named files); output compressed by -o extension. No more piping through gzip/zstd.
  • 💾 Scales past RAM-S external merge sort spills to disk and merges each chunk in parallel.
  • UX polish — sort-key highlighting (--color), a --progress bar/ETA, --stats, shell completions (--completions), and a generated man page (--man).

Why

GNU sort leaves performance on the table — historically single-threaded, and it pays a heavy Unicode-collation cost in non-C locales. xort is parallel by default and compares bytes by default (equivalent to LC_ALL=C), which is both fast and predictable.

Measured with hyperfine against GNU coreutils sort 9.4. To be fair, both tools run under LC_ALL=C and use all cores — we deliberately do not exploit GNU sort's much larger slowdown in a UTF-8 locale. Output is verified byte-identical to GNU.

Headline in-memory sorts (10M ints / decimals, 8M text lines, 4 cores):

Workload (input) GNU sort xort Speedup
Numeric, 10M ints (-n) 4.81 s 2.03 s 2.37×
Float, 10M decimals (-n) 4.92 s 2.32 s 2.12×
Text, 8M lines 2.15 s 1.23 s 1.74×
Unique text, 8M lines (-u) 2.54 s 1.54 s 1.65×
Top-100, 10M ints (--top 100) 2.12 s 0.64 s 3.33×

These wins hold across input sizes (1M–100M) and data distributions (uniform / sorted / reverse / nearly-sorted / low-cardinality); xort scales ~2.4× from 1→4 cores, and for structured data it sorts JSONL by a field faster than jq -s 'sort_by(…)'. The full multi-axis tables — size-scaling, distribution, thread-scaling, external (>RAM), field-keys, and structured — live in benchmarks/results.md, generated by scripts/benchmark.sh (QUICK=1 for a fast pass).

Two paths that started behind GNU have caught up: the external -S (>RAM) merge sorts each chunk in parallel (~1.3× faster than sort -S at the same budget on 10M ints), and field-key sorts (-k/-t) use decorate-sort-undecorate to extract every key once up front instead of on each comparison. A single key is now ~1.8× faster than GNU sort -k; multi-key sorts (e.g. -k2,2 -k3,3n) land roughly on par with GNU (up from ~1.3× slower), since chasing several precomputed keys per record trades parse cost for cache misses. results.md has the per-case numbers.

Global numeric sort (-n) takes an integer radix fast path: when every line's value is an exact i64, a stable LSD radix replaces the comparison sort. Measured on 10M integers (4 cores, LC_ALL=C), it cuts -n from 3.41 s → 2.46 s on wide-range (mostly unique) data and 2.39 s → 1.77 s on low-cardinality data — about 1.4× faster than xort's own comparison path, and ~2.5× faster than GNU sort -n (6.22 s / 3.99 s in the same run). Output stays byte-identical to GNU (value order, whole-line tie-break, and + treated as non-numeric like GNU — true on both the radix and the comparison paths); a fractional value or an out-of-i64-range magnitude transparently falls back to the arbitrary-precision comparison path.

Correctness is independently checked against GNU sort by scripts/difftest.sh.

How xort compares

GNU sort isn't the only game in town anymore — uutils/coreutils is a Rust rewrite of the whole coreutils suite (Ubuntu 26.04 ships it as the default sort). Its goal is GNU-compatible flags and behavior, not new capability, so it's a fair baseline for "what does xort add on top of a GNU-equivalent sort":

GNU sort uutils sort xort
Byte-identical to LC_ALL=C sort mostly¹ ✅ (155-case difftest)
Integer radix fast path for -n ✅ (~2.5× GNU)
--top N (bounded top-K, no full sort)
--count (sort | uniq -c fused)
--header (keep header row out of the sort)
--csv/--tsv, sort by column name
--json/--jsonl, sort by .path.field
--date-sort (ISO-8601/epoch/syslog)
Transparent gzip/zstd in/out
External (>RAM) merge sort (-S) ✅, chunk-parallel

¹ uutils targets full GNU compatibility but is still closing gaps (tracked upstream); xort's compatibility is pinned by its own 155-case differential test against real GNU sort output, run on every CI build. Both GNU sort and uutils sort support --parallel and default it to your core count, so xort's edge in the benchmark table above (measured against GNU with both using every core) isn't "parallel vs. not" — it's the radix fast path for -n and general implementation efficiency. We haven't published head-to-head xort-vs-uutils numbers yet; treat the feature rows above as the current comparison and the speed claims as GNU-only until we do.

In short: uutils re-implements GNU sort in Rust with GNU compatibility as the goal; xort starts from the same GNU-compatible baseline (byte-identical output, differentially tested) and adds the things GNU sort was never going to grow — bounded top-K, fused dedup+count, structured-data awareness, and date-aware ordering — while also sorting faster.

Install

cargo install xort       # from crates.io
cargo install --path .   # from a checkout
brew install OTL/xort/xort   # macOS / Linux, prebuilt binary, no Rust toolchain

Usage

xort accepts the common GNU sort flags:

xort [FILE...]            sort lines of text (stdin if no files)

  -n, --numeric-sort       compare by leading numeric value
  -g, --general-numeric    compare by general float value (incl. exponents)
  -h, --human-numeric      compare human sizes (2K, 1G, ...)
  -V, --version-sort       natural/version ordering (v2 < v10)
  -M, --month-sort         (unknown) < JAN < ... < DEC
  -k, --key=KEYDEF         sort by a key: F[.C][OPTS][,F[.C][OPTS]]
  -t, --field-separator=SEP
  -r, --reverse            reverse the result
  -u, --unique             output only the first of each equal-key run
  -s, --stable             keep input order among equal keys
  -f, --ignore-case        fold lower case to upper case
  -b, --ignore-leading-blanks
  -m, --merge              merge already-sorted inputs
  -c, --check              check whether input is sorted; rich diagnostics
  -z, --zero-terminated    lines end with NUL, not newline
  -o, --output=FILE        write result to FILE
  -S, --buffer-size=SIZE   spill to disk above SIZE (external merge sort)
  -T, --temporary-directory=DIR
      --parallel=N         use N threads

New in xort:
      --top=N              emit only the first N lines in sort order, using a
                           bounded selection instead of a full sort + head
      --count              with grouping, prefix each line with its count
                           (built-in `sort | uniq -c`)
      --header             keep the first line on top, exclude it from sorting
      --date-sort          sort by parsed date/time (ISO-8601, Unix epoch,
                           Apache/syslog logs); also a `-k` option letter `D`
      --csv / --tsv        parse quoted CSV/TSV; sort by column index or name
      --json / --jsonl     sort a JSON array / JSONL stream by a field path
      --color=WHEN         highlight the sort key (auto|always|never)
      --progress           show a progress bar / ETA on stderr (TTY only)
      --stats              print line counts and elapsed time to stderr
      --completions=SHELL  print a shell completion script
      --man                print the man page

Input compressed with gzip or zstd is decompressed transparently (detected by magic bytes, so file names need not match, and stdin works too), and output is compressed when -o names a .gz/.zst file — no piping through gzip/zstd required.

Examples

xort -n data.txt                      # numeric sort
xort -u names.txt                     # sorted unique
xort -n --top 10 metrics.txt          # 10 smallest, far cheaper than sort | head
du -b * | xort -nr --top 5            # 5 largest
xort --count access.log               # built-in `sort | uniq -c`, ~30x faster idiom
xort -t: -k3,3n /etc/passwd           # sort by 3rd colon field, numeric
xort --csv --header -k age -n people.csv   # CSV: sort by the "age" column
cat events.jsonl | xort --jsonl -k .ts     # JSONL: sort by .ts field
xort -n -S 256M huge.txt              # external merge sort for >RAM inputs
xort -n big.txt.gz -o sorted.zst      # gzip in, zstd out, decided by extension
xort --date-sort access.log           # order log lines chronologically
xort -k1,1D --progress events.tsv     # date key on field 1, with a progress bar

Compatibility

xort's default ordering is a byte comparison (LC_ALL=C semantics), so its output matches LC_ALL=C sort. The differential test (scripts/difftest.sh) checks xort against GNU sort across random word/number inputs and flag combinations.

Development

cargo test --all                 # unit + integration tests
cargo clippy --all-targets -- -D warnings
bash scripts/difftest.sh         # diff against GNU sort (requires `sort`)

License

MIT.

About

⚡ Fast, parallel drop-in replacement for Unix sort, in Rust

Topics

Resources

Code of conduct

Contributing

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages