Skip to content

Single-read multi-product tile jobs (F3 core) - #75

Draft
dshean wants to merge 4 commits into
pr/batch-driverfrom
pr/f3-core
Draft

Single-read multi-product tile jobs (F3 core)#75
dshean wants to merge 4 commits into
pr/batch-driverfrom
pr/f3-core

Conversation

@dshean

@dshean dshean commented Jul 17, 2026

Copy link
Copy Markdown
Member

Stack 3/8. F3 core: single-read multi-product tile jobs.

Commits: 2d8b083 (single-read tile jobs for all products), fc6f869 (local-LAZ path onto the shared builder), a66136c (EPT source CRS stamped on tile cache round-trip), 420a35b (empty-tile handling + no-data/empty-mosaic guards).

User-visible / BREAKING for Python-API users: create_ept_3dep_pipeline/create_lpc_pipeline now return list[dict] tile jobs (was a 4-tuple of per-product pipeline lists) and gain products=; --products accepts a free comma list (dsm,dtm_no_fill,dtm_fill,intensity; all/dtm aliases kept — all main-era CLI values still valid). CLI behavior otherwise unchanged; one EPT read per tile instead of four (the headline speedup).

Validation: the engine under every f3 production run; overlap dz +0.02–0.04 m vs independent-era surveys.

Merge with a merge commit.

🤖 Generated with Claude Code

dshean and others added 4 commits July 5, 2026 19:53
- Read EPT points once per tile into a LAS 1.4/pf6 cache LAZ, then emit
  every requested product from local executions with chained writers
  (was: one full EPT fetch per product per tile, ~4x redundant transfer;
  PDAL cannot branch one reader into multiple writer leaves, so the job
  is a fetch step plus per-chain linear pipelines)
- Product registry + chain-signature grouping: products with identical
  built filter chains share one execution (dsm+intensity under first_idw;
  the DTM pair always); a writer-only future product merges at zero extra
  reads; pf6 cache keeps PointSourceId/GpsTime as the per-lift seam
- products accepts comma-separated subsets ("dsm,intensity"; "dtm" alias
  for both DTM variants; "all" default); unrequested products are no
  longer built
- One dask task per tile replaces four per-product batches; per-execution
  resume (skip iff ALL outputs valid); cache never trusted on resume and
  unlinked in-task; retry/backoff preserved per pipeline
- Local-LAZ input unchanged: legacy per-product pipelines adapted into
  degenerate single-execution tile jobs
- Tile output filenames byte-identical to the per-product path (resume,
  mosaic and cleanup compatibility); downstream mosaics/warps keyed on
  the requested product set
- 9 offline tests: job structure (first_idw+hag, n-pct, subsets),
  legacy-vs-consolidated raster equivalence, resume granularity, fetch
  failure, empty-branch behavior

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- create_lpc_pipeline now returns tile jobs from create_tile_pipelines
  (issue #14 direction: one reader-agnostic core builder for EPT and
  local input), replacing its four per-product pipeline copies
- In-pipeline reprojection injected once via extra_pipeline_kwargs and
  shared by every product chain, so dsm+intensity and the DTM pair merge
  into chained-writer executions: 2 local reads per tile instead of 4
- save_pointcloud preserved as a chained writers.las before the DSM
  writer (legacy behavior verbatim, including the historical .laz.laz
  double-extension filename); only emitted when dsm is requested
- No cache step for local input (files are cheap to re-read)
- Commit-1 legacy adapter removed; rasterize has one executor code path
- New offline test: LPC job structure, reprojection stage, .laz.laz
  save writer, end-to-end execution, dsm-less subset

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The product raster CRS depended entirely on the LAS-header SRS surviving
the cache write/read round-trip. On another environment's PDAL/GDAL build
the SRS did not reconstruct on read, so writers.gdal wrote CRS-less
rasters and every tile failed check_raster_validity (SF rerun aborted,
100% of tiles invalid); this env reconstructs it fine, which is why the
UW-campus A/B and local SF repros passed.

Make it deterministic: thread the EPT's declared SRS (POINTCLOUD_CRS[i],
from ept.json) into create_tile_pipelines and stamp it on both ends of the
cache round-trip -- a_srs on the cache writers.las and override_srs on the
cache readers.las -- so tiles carry the CRS regardless of LAS-header
round-trip behavior. Non-cached (single-product, reader inlined) and LPC
(in-pipeline reprojection) paths are unaffected. _pdal_srs_string renders
an EPSG code when available, else WKT.

Tests: assert the stamps appear in the generated pipelines; reproduce the
failure with a deliberately untagged cache LAZ (CRS-less invalid raster)
and prove override_srs recovers the CRS. Verified live on a tiny SF AOI
via rasterize-projects: 3/3 all products valid, tiles EPSG:3857, mosaic
UTM 10N G2139.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Empty tiles (a survey does not cover part of the shared AOI) previously
each wrote a full-size CRS-less nodata raster, logged a spurious
"invalid raster" ERROR, and counted as a failed tile -- flooding the log
and making the final tally unreadable (e.g. "190/340" with 150 merely
empty). And a project with zero coverage over the AOI crashed with
"list index out of range" at the single-tile rename, while an empty
mosaic list failed cryptically ("vrt: No such file or directory").

- execute_tile_job now returns {"empty": bool, "outputs": {...}}. The fetch
  point count (via pipeline.execute()) detects a 0-point tile: skip the
  product executions, write nothing, log no ERROR, report empty.
- rasterize tracks empties separately from failures: "N/M tiles had no
  points (survey does not cover them)" instead of counting them failed;
  "failed" is now measured against data-bearing tiles only.
- No-data guard: if no product has any valid tile (0 readers or all-empty),
  skip mosaic/reprojection/stamping and finish with run_status completed +
  a "no data" note (driver marks the project completed, not failed) instead
  of an IndexError.
- raster_mosaic raises a clear ValueError on an empty input list instead of
  the cryptic downstream VRT error.

Verified on a mixed land/water SF AOI via rasterize-projects: covered
project completes with valid mosaics; a non-covering project finishes
cleanly as no-data. Tests: F5-lite empty-tile (empty=True, no rasters,
no ERROR); existing execute_tile_job tests updated to the new return shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant