Build one FASTQ-to-analysis-ready BAM path with fused QC #5

Goal

Build Rminibwa's production path from FASTQ to one analysis-ready alignment bundle:

1. read, decompress, validate, and map each FASTQ record once;
2. make deterministic template-aware duplicate decisions without writing a draft alignment file;
3. coordinate-sort with bounded spill;
4. write one final BAM while building BAI or CSI; and
5. collect QC metrics from the same native operation, without rescanning the completed BAM.

Rminibwa continues to ship the official minibwa CLI unchanged. A separate rminibwa CLI, shell-pipe protocol, Arrow IPC writer, and mandatory public alignment relation are not part of this work.

"One pass" means the FASTQ inputs are parsed and mapped once. Coordinate sorting is an explicit materialization step. Query-order metrics may be accumulated before sorting; alignment, duplication, insert-size, coverage, and index metrics consume the final coordinate-ordered records once.

Package responsibilities

Rminibwa owns FASTQ validation, alignment orchestration, template accounting, duplicate decisions, sorting, QC collection, artifact staging, and bundle validation.

Rduckhts is Rminibwa's sole htslib build provider. Configure must consume and validate Rduckhts::rduckhts_htslib_config(), compile and link only against the exact headers and library named by that receipt, and prove the same htslib identity at runtime. Rduckhts remains responsible for that htslib build, standard readers/indexers, runtime plugin setup, and cross-format validation.

The official packaged minibwa executable remains an upstream compatibility surface. The production BAM/QC operation is native package functionality and must not fork or wrap a second CLI implementation of the aligner.

Native producer contract

Use bounded pure-C batches/arenas that preserve:

• original sequence and quality bytes;
• every primary, secondary, supplementary, and unmapped record;
• packed CIGAR, flags, coordinates, MAPQ, mate fields, template length, and typed hot tags (AS, NM, MQ, MC, RG);
• exact @SQ, caller-supplied @RG, and @PG facts;
• stable template/input order and a unique deterministic record tie key; and
• typed EOF, malformed-input, validation, cancellation, allocation, sort, writer, index, and collector failures.

Complete QNAME groups use the explicit uniqueness policy tracked in #4. Parallel callback order is never treated as record order.

Execution shape

FASTQ
  -> strict paired/single-end validation
  -> native query-group mapping batches
       |- input/read/base accounting
       `- template facts needed by duplicate marking
  -> compact records plus deterministic coordinate/tie keys
  -> bounded coordinate sort with spill
  -> one final ordered-record traversal
       |- finalize duplicate flags
       |- write BAM through the Rduckhts-supplied htslib
       |- build BAI or CSI from final virtual offsets
       |- alignment and mapping-quality QC
       |- duplication QC
       |- insert-size and orientation QC
       |- per-reference, fixed-bin, and requested target coverage
       `- output/accounting receipts
  -> close, validate, and publish the complete bundle

The sorted carrier may use DuckDB vectors and ORDER BY, or another measured bounded sorter. That is an implementation choice, not the product. If DuckDB is used, volatile exact-version APIs stay in a narrow adapter and every supported artifact must pass real load/run conformance tests.

No SAM, draft BAM, persistent alignment table, Arrow conversion, or per-record R callback belongs in the required path. Optional relational inspection can be added after this bundle is correct and competitive; it must reuse the same producer rather than redefine alignment semantics.

QC contract

The first vertical slice emits clearly defined metrics for:

• accepted/rejected FASTQ reads and templates;
• total, primary, secondary, supplementary, mapped, and unmapped records;
• paired, properly paired, duplicate, read-group, and library accounting;
• read/base quality and mapping-quality summaries;
• insert-size and pair-orientation distributions;
• per-reference counts; and
• fixed-bin and configured target coverage when requested.

Each metric states whether it counts reads, records, templates, fragments, aligned bases, bins, or intervals and whether it is measured before or after filtering and duplicate marking. Small outputs may be typed TSV/DuckDB tables; large coverage outputs may use Parquet or indexed BGZF text only after format-specific benchmarks. Output format choices must not introduce another alignment scan.

Riker and pinned Picard/fgbio tools are semantic and differential baselines, not subprocesses or runtime dependencies.

Artifact and failure contract

A successful operation publishes at least:

sample.bam
sample.bam.bai or sample.bam.csi
sample.analysis/manifest.json
sample.analysis/qc-*.tsv (or an explicitly versioned typed equivalent)

All outputs are created under one unique staging directory on the destination filesystem. The operation must close every BAM/index/QC handle, validate coordinate order and accounting, exercise index queries, and then rename completed outputs. A final completion marker is published only after the native operation returns successfully and the R wrapper revalidates the bundle.

Any source, mapping, sorting, spill, disk, BGZF, index, collector, cancellation, or finalization failure removes or quarantines staged files idempotently. A partial BAM or complete-looking QC bundle is never reported as success.

Resource contract

One operation accepts explicit total threads, resident-memory limit, scratch directory and optional disk allowance, cancellation token, BAM compression level, and BAI/CSI policy. The planner divides these resources across FASTQ decompression, alignment, sort, htslib/BGZF, queues, and collectors. Components must not each claim the full budget, and no behavior-changing hidden environment variables are used.

Acceptance criteria

• Configure and runtime checks fail closed on a missing or mismatched Rduckhts htslib receipt.
• Real single- and paired-end FASTQ fixtures produce deterministic coordinate-sorted BAM plus BAI/CSI.
• The BAM validates with htslib/samtools and Rduckhts readers, metadata inspection, and region queries.
• Fixtures cover mapped/unmapped mates, clipping/indels, secondary/supplementary records, multiple read groups/libraries, duplicates, deterministic ties, empty input, malformed/truncated FASTQ, cancellation, and forced spill.
• FASTQ, mapped, sorted, duplicate, BAM-written, index-observed, and every QC total reconcile.
• Duplicate partitions and metrics are compared with pinned samblaster/Picard/fgbio or Riker baselines; intentional differences are documented.
• Failure injection proves that no completed-looking bundle survives source, sort, sink, disk, index, collector, finalize, or interruption failures.
• The official packaged minibwa CLI remains unchanged apart from independently justified upstream synchronization.

Benchmark evidence

Under identical total threads, memory, scratch storage, input, ordering, BAM compression, and index policy, report wall time, CPU time, peak RSS, scratch bytes, durable bytes, records/s, and per-stage thread allocation for:

• the fused FASTQ -> sorted/indexed BAM + QC operation;
• official minibwa -> samtools sort/index -> equivalent QC tools;
• DuckDB sorting versus samtools sorting and any custom sorter; and
• one and eight total threads, including forced spill.

The fused path must be compared with the complete equivalent pipeline, not bare alignment or BAM writing.

Out of scope

CRAM, Arrow C Stream/IPC, a new CLI, generic columnar alignment export, variant calling, phasing, annotation, broad provider catalogs, and an external collector/plugin ABI are separate work. This issue establishes the analysis-ready BAM and fused-QC product.