Single-read multi-product tile jobs (F3 core) - #75
Draft
dshean wants to merge 4 commits into
Draft
Conversation
- Read EPT points once per tile into a LAS 1.4/pf6 cache LAZ, then emit
every requested product from local executions with chained writers
(was: one full EPT fetch per product per tile, ~4x redundant transfer;
PDAL cannot branch one reader into multiple writer leaves, so the job
is a fetch step plus per-chain linear pipelines)
- Product registry + chain-signature grouping: products with identical
built filter chains share one execution (dsm+intensity under first_idw;
the DTM pair always); a writer-only future product merges at zero extra
reads; pf6 cache keeps PointSourceId/GpsTime as the per-lift seam
- products accepts comma-separated subsets ("dsm,intensity"; "dtm" alias
for both DTM variants; "all" default); unrequested products are no
longer built
- One dask task per tile replaces four per-product batches; per-execution
resume (skip iff ALL outputs valid); cache never trusted on resume and
unlinked in-task; retry/backoff preserved per pipeline
- Local-LAZ input unchanged: legacy per-product pipelines adapted into
degenerate single-execution tile jobs
- Tile output filenames byte-identical to the per-product path (resume,
mosaic and cleanup compatibility); downstream mosaics/warps keyed on
the requested product set
- 9 offline tests: job structure (first_idw+hag, n-pct, subsets),
legacy-vs-consolidated raster equivalence, resume granularity, fetch
failure, empty-branch behavior
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- create_lpc_pipeline now returns tile jobs from create_tile_pipelines (issue #14 direction: one reader-agnostic core builder for EPT and local input), replacing its four per-product pipeline copies - In-pipeline reprojection injected once via extra_pipeline_kwargs and shared by every product chain, so dsm+intensity and the DTM pair merge into chained-writer executions: 2 local reads per tile instead of 4 - save_pointcloud preserved as a chained writers.las before the DSM writer (legacy behavior verbatim, including the historical .laz.laz double-extension filename); only emitted when dsm is requested - No cache step for local input (files are cheap to re-read) - Commit-1 legacy adapter removed; rasterize has one executor code path - New offline test: LPC job structure, reprojection stage, .laz.laz save writer, end-to-end execution, dsm-less subset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The product raster CRS depended entirely on the LAS-header SRS surviving the cache write/read round-trip. On another environment's PDAL/GDAL build the SRS did not reconstruct on read, so writers.gdal wrote CRS-less rasters and every tile failed check_raster_validity (SF rerun aborted, 100% of tiles invalid); this env reconstructs it fine, which is why the UW-campus A/B and local SF repros passed. Make it deterministic: thread the EPT's declared SRS (POINTCLOUD_CRS[i], from ept.json) into create_tile_pipelines and stamp it on both ends of the cache round-trip -- a_srs on the cache writers.las and override_srs on the cache readers.las -- so tiles carry the CRS regardless of LAS-header round-trip behavior. Non-cached (single-product, reader inlined) and LPC (in-pipeline reprojection) paths are unaffected. _pdal_srs_string renders an EPSG code when available, else WKT. Tests: assert the stamps appear in the generated pipelines; reproduce the failure with a deliberately untagged cache LAZ (CRS-less invalid raster) and prove override_srs recovers the CRS. Verified live on a tiny SF AOI via rasterize-projects: 3/3 all products valid, tiles EPSG:3857, mosaic UTM 10N G2139. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Empty tiles (a survey does not cover part of the shared AOI) previously
each wrote a full-size CRS-less nodata raster, logged a spurious
"invalid raster" ERROR, and counted as a failed tile -- flooding the log
and making the final tally unreadable (e.g. "190/340" with 150 merely
empty). And a project with zero coverage over the AOI crashed with
"list index out of range" at the single-tile rename, while an empty
mosaic list failed cryptically ("vrt: No such file or directory").
- execute_tile_job now returns {"empty": bool, "outputs": {...}}. The fetch
point count (via pipeline.execute()) detects a 0-point tile: skip the
product executions, write nothing, log no ERROR, report empty.
- rasterize tracks empties separately from failures: "N/M tiles had no
points (survey does not cover them)" instead of counting them failed;
"failed" is now measured against data-bearing tiles only.
- No-data guard: if no product has any valid tile (0 readers or all-empty),
skip mosaic/reprojection/stamping and finish with run_status completed +
a "no data" note (driver marks the project completed, not failed) instead
of an IndexError.
- raster_mosaic raises a clear ValueError on an empty input list instead of
the cryptic downstream VRT error.
Verified on a mixed land/water SF AOI via rasterize-projects: covered
project completes with valid mosaics; a non-covering project finishes
cleanly as no-data. Tests: F5-lite empty-tile (empty=True, no rasters,
no ERROR); existing execute_tile_job tests updated to the new return shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3/8. F3 core: single-read multi-product tile jobs.
Commits: 2d8b083 (single-read tile jobs for all products), fc6f869 (local-LAZ path onto the shared builder), a66136c (EPT source CRS stamped on tile cache round-trip), 420a35b (empty-tile handling + no-data/empty-mosaic guards).
User-visible / BREAKING for Python-API users:
create_ept_3dep_pipeline/create_lpc_pipelinenow returnlist[dict]tile jobs (was a 4-tuple of per-product pipeline lists) and gainproducts=;--productsaccepts a free comma list (dsm,dtm_no_fill,dtm_fill,intensity;all/dtmaliases kept — all main-era CLI values still valid). CLI behavior otherwise unchanged; one EPT read per tile instead of four (the headline speedup).Validation: the engine under every f3 production run; overlap dz +0.02–0.04 m vs independent-era surveys.
Merge with a merge commit.
🤖 Generated with Claude Code