threejs-cf-zxy-s1m

A CONUS flight-simulator-style streaming viewer: NAIP aerial imagery and USGS 3DEP S1M 1-meter terrain, normalized server-side onto a single EPSG:3857 z/x/y quadtree and streamed to a Three.js client through CloudFront CDN.

_{In flight at 43.527 N, −109.703 W — 8,019 m, heading 341° — over the upper Wind River country, Wyoming. NAIP aerial imagery draped on 1 m S1M terrain, both streamed as z/x/y WebP tiles. Vertical exaggeration is 1.5×, the panel's default, so relief reads a bit steeper than truth. The counters are a live flight, not a cold load: 1,028 tiles prefetched along the flight vector, and 1,463 of 2,570 tile textures served by recycling rather than allocation.}

What's built today: a camera you can fly over 1-meter S1M terrain, starting at the Wyoming area S1M tile group, with server tiles end-to-end. Everything below — the architecture, the tile contracts, and how to run it — is self-contained; FLIGHT-SIM-PLAN.md holds the longer-form design record and the phases still ahead, and is worth reading only if you want the reasoning behind a decision rather than the decision itself.

Its sibling project, deckgl-s3-cog-s1m, reads Cloud-Optimized GeoTIFFs directly in the browser over HTTP range requests — no tile server, no terrain server. That architecture is excellent for inspecting a viewport: you see exactly the source pixels, and the only backend is a small index lookup.

But that is a poor fit for flight. A camera moving at 800 knots crosses tile boundaries several times a second, and every new COG means fresh range reads, fresh TIFF decode, and a per-source CRS warp on the client — all while the frame budget is 16 ms. The client also has to understand every source's quirks: NAIP's requester-pays bucket, S1M's Albers projection, the 1/3-arcsecond fallback where 1-meter coverage hasn't landed yet.

So this repository inverts the design. Every source — 1 m or 10 m DEM, COG mosaic or proxied pyramid — is normalized server-side onto the same 512 px z/x/y grid and encoding, so the client speaks exactly one tile contract and never learns which source produced a tile. Warm tiles are CloudFront edge hits; only misses invoke Lambda. The browser gets to focus on rendering.

The trade is explicit: you give up direct-from-source pixels and accept a render tier, in exchange for a client simple enough to fly.

Part of the interest in building this was to find out whether anything we built into pTolemy3D over twenty years ago still holds up. That viewer solved streaming-terrain flight in Java and JOGL, against JPEG2000 imagery and a very different web back then — but the hard parts of flying a camera over tiles have not changed much.

Several of its ideas came across intact: a background loading pipeline so the render loop never stalls, nearest-tile-first scheduling so the closest terrain wins the next download slot, aborting requests for tiles that left the view, and altitude-scaled movement speed. Others did not survive contact — its view-cone logic is frustum culling, not LOD, and an early attempt to scale subdivision by camera pitch here caused LOD inversions and was replaced by screen-space error.

README-PTOLEMY3D.md traces each pattern from the original implementation to where it now lives in client/src/core/.

Good question, since both exist and this repo leans on rio-tiler for the actual reads.

TiTiler is a more capable tiler than this one and would be the right answer for a general-purpose service. Its flexibility is the problem here: its endpoints are configured through query strings (?rescale=, ?bidx=, ?expression=, ?colormap=), and this design needs the opposite — the URL is the cache key. Every rendering decision is frozen server-side into a fixed contract (512 px, Terrarium, lossless WebP for elevation) so CloudFront can forward no query strings at all and every viewer shares one immutable object per tile. Adopting TiTiler would mean locking down most of what makes it worth adopting.

cogeo-mosaic is the closer call, and it would do this job perfectly well — the asset set changes as a periodic batch job, not a stream, so a MosaicJSON regenerated on the same cadence would track it fine.

The reason it isn't here is that a published STAC-GeoParquet index already answers "which COGs intersect this tile." This repo reads collection=naip-visualization from s3://naip-geoparquet-index/manifest-index and does not write it, and querying it directly is one ~20-line SQL builder (resolver.py). MosaicJSON would mean deriving a second index from an upstream this repo doesn't control, then keeping the two in step forever. One index is simpler than two.

Worth knowing before you fork, because "maintained" is doing two jobs in that sentence. The data is maintained: the ingest pipeline in deckgl-s3-cog-s1m is what scans the naip-visualization bucket and builds the GeoParquet lake behind this collection. The publication step is not — projecting that lake into the STAC index this tiler reads was done once, in July 2026, by an ad-hoc DuckDB COPY that lives in no repository and that nothing re-runs. New NAIP vintages reaching the lake would not reach this index on their own.

Not simpler code, though, which is worth saying plainly: DuckDB rides along in the Lambda, and the SQL is mine to maintain. cogeo-mosaic is less code and far better tested.

Scale is the other half of it, and it cuts the other way. A MosaicJSON is one document mapping quadkeys to assets, and CONUS NAIP runs to roughly 220k quarter quads per coverage year — indexed at z12 that is a document of tens of megabytes per vintage, fetched and parsed to answer one tile. cogeo-mosaic's answer is a DynamoDB or SQLite backend, which works; but at that point the choice is between two databases rather than between a database and a file, and the less-code argument has spent itself. Parquet stays flat here: row-group statistics mean only the groups whose bbox overlaps the tile are ever read, so the index grows without the per-tile read growing with it.

So the verdict is scale-dependent rather than absolute. For a project-sized asset set MosaicJSON wins, and wins easily. For CONUS, with an index already published next door, this way does.

PostGIS deserves its own mention, because it is the obvious answer and would have been the easy one: put the footprints in a table, add a GiST index, and "which COGs intersect this tile" is a one-liner someone else has already optimised over twenty years.

The reason it isn't here is less architectural than it sounds. This is a prototype that sits idle for weeks between the afternoons anyone looks at it, and I did not want a server-side stack to look after.

That argument is weaker than it used to be, and it is worth being accurate about: Aurora Serverless v2 has scaled to zero ACUs since late 2024, so a paused PostGIS cluster bills storage and nothing else. It parks itself. What rules it out here is the wake-up — resuming from zero takes on the order of 15 seconds, and the query it would serve sits on the cold-tile path with the browser waiting. Avoiding that means never letting it pause, which is the cost back again. There is also a neat catch: RDS Proxy, the standard answer to a Lambda opening connections to Postgres, is on the list of configurations that disable auto-pause. Connection pooling and scale-to-zero are mutually exclusive.

DuckDB sidesteps the question rather than answering it. The index is a file on S3, so there is nothing to wake, and compute exists only for as long as a cold tile takes to build.

sequenceDiagram
    autonumber
    participant B as Browser (Three.js)
    participant CF as CloudFront
    participant L as Tiler Lambda
    participant S3 as S3 COGs / upstream caches

    B->>CF: GET /terrain/15/9633/12332.webp
    alt warm tile — the common case
        CF-->>B: 512 px WebP straight from the edge
    else cold tile
        CF->>L: signed origin request (OAC)
        L->>L: DuckDB: which COGs cover this tile?
        L->>S3: range reads
        S3-->>L: COG blocks
        L->>L: mosaic → warp → Terrarium → WebP
        L-->>CF: 512 px WebP, immutable
        CF-->>B: tile (edge is now warm)
    end
    B->>B: decode → heightfield mesh + texture

1. One tile contract. The client requests 512 px imagery and elevation over a single XYZ Web-Mercator quadtree. It never knows which source produced a tile (beyond the X-DEM-Source debug header).
2. Server-side normalization. At high zoom the Lambda builds tiles from source COGs — locating intersecting files via a DuckDB GeoParquet index, then mosaicking and warping them onto the tile grid. At low zoom it skips the COGs and proxies pre-rendered pyramids instead.
3. The browser only ever talks to CloudFront. Warm tiles are edge-cache hits (path-only keys, immutable); only misses invoke the Lambda.

Per zoom band, the source data is:

That split is a scoping decision, not a ranking of quality. What this demo set out to test is the top of the table — 1-meter S1M terrain with NAIP at native resolution, roughly z15–20, where the tile count explodes and the streaming either works or visibly doesn't. Everything below is the approach to it: context you pass through on the way down, where a 10 m DEM and a rendered basemap are indistinguishable from something better. Those bands therefore lean on whatever already exists — an external image service at low-zoom imagery, a global Terrarium pyramid under z11 — rather than being built out.

For imagery there is a mechanical reason too, not just a priority. NAIP ships as quarter-quad COGs, roughly 5 x 7 km each, so the number a single tile overlaps grows fourfold every zoom out: fine at z14, but a z9 tile spans about 119 of them against a 64-asset cap. Below z14 the mosaic stops being the cheaper way to answer, which is why that band is a single rendered request to an image service instead.

_{Buildings are the one layer the database emits in its final wire format: ST_AsMVT assembles the vector tile inside DuckDB, and the client extrudes the footprints onto terrain it already has.}

Elevation ships as Terrarium-encoded Terrain-RGB in lossless WebP, the same packing elevation-tiles-prod already uses. That makes the far-field path a pure 2×2 pixel copy — no decode, no re-encode, and no conversion bugs at the zoom boundary where near-field S1M meets the global pyramid. The client runs exactly one decoder everywhere. Imagery ships as lossy WebP (q≈75).

Tiles are served max-age=31536000, immutable, so a bad tile is a permanent bad tile. The render path is written around that: a source read that fails transiently (network, 5xx, throttling) returns 503 with Retry-After so the client retries, while only a genuinely absent source falls back to sea level or a coarser DEM. Unknown failure modes fail toward 503 rather than baking a hole into a tile that can never be evicted. Endpoints also 404 above their source's native resolution, so the CDN never caches upsampled junk over an unbounded key space.

Layer and year live in the path, never a query string, and the CloudFront cache policy forwards no query strings, cookies, or headers. Every viewer therefore shares one cached object per tile. Origin Shield funnels all POPs through one regional cache, so a cold tile costs at most one Lambda render globally instead of one per POP. The dev access key rides in a query param precisely because it is outside the cache key — rotating it needs no invalidation.

The client refines a tile quadtree while a tile's geometric error projects to more than a pixel threshold on screen — monotonic in distance, so LOD rings can never invert. Server tiles carry no overlap ring; seam hiding is entirely the client's job, via one extra vertex ring per tile dropped by a per-zoom skirt height that handles both same-zoom cracks and cross-LOD T-junctions.

A Mercator meter is not a ground meter. Every true-meters → world conversion routes through sec(lat), so slopes don't flatten going north. Positions are emitted relative to each tile's NW anchor as float32 — raw Mercator X for CONUS is ~1e7, where float32 resolution is ~1 m, useless against 30 cm imagery.

Fetch, decode, and mesh building run in a Web Worker pool with priority queueing, abort-on-cancel, and velocity-vector prefetch along the flight path. Tile URL construction and imagery-source routing live in one shared module so the worker (production) and the main-thread fallback (what the tests exercise) can never drift apart.

• GET /imagery/{layer}/{year}/{z}/{x}/{y}.webp — 512 px WebP q≈75, path-only cache keys, per-layer maxzoom from registry gsd.
• GET /terrain/{z}/{x}/{y}.webp — 512 px Terrarium-encoded Terrain-RGB, lossless WebP, vanilla registration (no overlap); the client builds skirts.
• GET /basemap/{z}/{x}/{y}.webp — low-zoom 512 px imagery rendered from the USGS NAIP ImageServer (the browser never hits ArcGIS directly).
• GET /footprints/{s1m,usgs13}.json — static gzipped GeoJSON of DEM COG footprints, served straight from S3 (no Lambda); rebuilt by tiler/scripts/build_footprints.py.
• GET /buildings/{z}/{x}/{y}.pbf — MVT building footprints and heights from the Overture lake, extruded client-side. Served at z ≥ 14, 404 below.

All five are path-only by design. DEM band thresholds are tiler config (TILER_USGS_MIN_ZOOM / TILER_S1M_MIN_ZOOM), never per-request parameters — they change what a tile contains, which must not vary independently of the cache key.

This prototype stands on public data services it does not control, and two of them have taken the viewer down — within ten days of each other in August 2026. Both failures look like the app is broken. Neither is, so they are worth recognising on sight.

Overture retires the release the buildings manifest points at. /buildings resolves against a manifest naming which Overture GeoParquet row groups intersect a tile, and that manifest points into one specific release. Overture deletes old releases as new ones land, and then every building tile answers 404 "no building coverage" — Manhattan reports nothing while holding 2,700 of them.

Symptom: buildings vanish everywhere, at every zoom, and stay gone.

The imagery upstream goes down. /basemap renders from the USGS NAIP ImageServer at imagery.nationalmap.gov, which has returned 502s and 504s; the USDA service it replaced stopped completing TLS handshakes for about six days.

Symptom: ground renders flat green (fallbackColor, 0x556655) where imagery never arrived. Terrain, buildings and the DEM are unaffected — only the texture is missing.

That one degrades rather than breaks, deliberately: a failed imagery fetch no longer costs the tile its terrain, so you get flat ground rather than a hole with sky through it; tiles retry three times and then settle for the fallback instead of billing requester-pays reads against a dead upstream forever; and the HUD IMAGERY: line reads retrying N... then N tiles unavailable (upstream), so the cause is on screen rather than in a log.

Recovery for both — rebuilding the manifest, the two paths it must be published to, and how to confirm the upstream rather than guess — is in infra/README.md.

Needs Node 20+, Python 3.12+, Docker (for SAM container builds), and the AWS + SAM CLIs with credentials for anything cloud-side — NAIP is requester-pays, so reads are billed to the caller.

(cd tiler  && pip install -e ".[dev]")   # tiler: [dev] adds pytest + ruff
(cd client && npm install)               # client

The tiler's virtualenv stores absolute paths, so it does not survive the repo being moved or renamed; if pytest stops importing tiler, delete .venv and reinstall.

Run both locally — tiler on :8000, client on :5180:

(cd tiler  && uvicorn tiler.app:app --reload)   # one terminal
(cd client && npm run dev)                      # another

A word on what a fresh clone can and cannot do, because none of the three tile sources work out of the box. NAIP is requester-pays, so AWS credentials are a precondition for this project rather than an optional extra:

To bake that block:

(cd tiler && .venv/bin/python scripts/bake.py 40.48 -74.66 14 --grid 4)

It writes a 4×4 z14 block — 49 files, ~4 MB — into client/public/tiles/. That is a fixed block, not a world: it exists so the client can be worked on without paying tiler latency on every frame, not so you can fly. The viewer starts over Wyoming by default, so aim it at the block explicitly:

http://localhost:5180/?src=local&lat=40.48&lon=-74.66

Both suites are hermetic — S3 and HTTP are mocked, so no credentials and no network:

(cd client && npm run typecheck && npm test)
(cd tiler  && ruff check . && python -m pytest)

Deployment is two CDK stacks in us-west-2, the region holding the index and the source COGs. One script does all of it — build, both stacks, bucket seed, invalidation, and client/.env.local:

infra/deploy.sh

infra/README.md has everything else you need before pointing that at a real account: the one-time bootstrap, the colima DOCKER_HOST export, why the distribution deploys disabled, how the dev access key works, smoke-testing the IAM-authed Function URL, and what to read in cdk diff before deploying over live stacks. Index and asset builders are in tiler/scripts/.

• USGS 3DEP for the Seamless 1-meter DEM (S1M) and the 1/3-arcsecond DEMs, and USDA FPAC/APFO for NAIP — both published openly on S3.
• rio-tiler, GDAL, and the COG standard, which do the real work of turning range reads into tiles.
• DuckDB for in-process spatial queries over GeoParquet, removing the need for an always-on spatial database.
• Mapzen / Terrarium and the AWS-hosted elevation-tiles-prod pyramid for the global elevation fallback and its encoding.
• Three.js for the rendering layer.

This project is licensed under the MIT License.