Skip to content

docs: explorations — 3D from imagery, tile enhancement, neural textures, ML on WebGPU/wgpu (research notes) - #47

Open
valdo404 wants to merge 11 commits into
mainfrom
claude/3d-tiles-shape-transformation-syp59p
Open

valdo404 wants to merge 11 commits into
mainfrom
claude/3d-tiles-shape-transformation-syp59p

Conversation

@valdo404

@valdo404 valdo404 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What

Seven research and architecture notes in docs/explorations/, written in French like the other R&D docs. This is documentation only: no code, and nothing here is proposed for implementation.

Note Question
3d-from-imagery-tiles.md Can Tuile synthesize a credible 3D scene from the raster imagery tile + terrain tile it already drapes? Covers monocular height, shadow and off-nadir cues, LoD2, completion trained with holes, façades, vegetation, satellite→city generators, inverse procedural modeling, datasets and licences.
shape-from-photorealistic-tiles.md The mesh-based variant (photogrammetric tiles → segmentation / LoD2 / delighting), and the licensing blocker on commercial photorealistic tiles.
imagery-tile-enhancement.md Improving the imagery tiles themselves, as a complement to doc 18. It opens with what is actionable now: the viewer samples without anisotropic filtering, mips are averaged in sRGB, and reprojection is bilinear. It then covers SR, cross-zoom consistency, JPEG artifacts, colour, haze, clouds and IQA.
neural-texture-representations.md Neural texture compression (NTC and BC-latent variants), learned and overfitted codecs, INRs, 2D Gaussian images and embedding tiles. Each is judged against tile constraints.
webgpu-neural-networks.md Running neural networks on WebGPU/wgpu: platform support, runtimes, the Rust ecosystem, published trials, dispatch overhead, limits, kernels, numerics, and guidance and experiments for a Rust/wgpu engine.
onnx-webgpu-wgpu.md The ONNX path end to end (ORT-Web, the native Dawn EP, burn-onnx, ort/ort-web, tract, RTen, candle, wonnx, ONNX→WGSL compilers), with a recommendation per target.
gpu-tile-proxy.md Architecture: a GPU worker serving model-derived tile layers. To tuile-tile-server it is just another Upstream. The client always shows the raw tile and swaps it once the derived one is ready. Also covers the three tiers (browser, proxy, farm), determinism and provenance, licences, a cost model to fill by measurement, and open questions.

Facts checked directly in published sources

  • wgpu 29.0.3 cannot expose or adopt a JS GPUDevice through its public API. mod backend; is private (src/lib.rs:114) and WebDevice.inner is pub(crate) (src/backend/webgpu.rs:1202). As a result, JS ML runtimes cannot share Tuile's device on the web.
  • Burn/CubeCL track a specific wgpu version. On crates.io, cubecl-wgpu 0.10 (Burn 0.21) depends on wgpu ^29, which matches Tuile. cubecl-wgpu 0.11 (Burn 0.22) depends on wgpu ^30.

Why

These notes scope research directions for an open-source tile engine before any code is written: what the literature provides, which open data supports it, what licences rule out, and what is actionable with a few GPUs.

Caveats

  • Most references were not read in full text. The research environment could not reach arxiv.org, CVF, Copernicus, Hugging Face, developer.chrome.com or data.geopf.fr, so most references come from abstracts and snippets. Unconfirmed venues, numbers and licences are marked (non vérifié).
  • The PR is large: about 4,000 lines across seven documents. That is above the < ~800-line guidance in docs/20-conventions.md, which targets code PRs. I can split it if preferred.
  • No crate is touched, so cargo checks are unaffected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3

@valdo404 valdo404 changed the title docs: explorations — credible 3D from imagery tiles (literature review) docs: explorations — 3D from imagery tiles, and imagery tile enhancement (literature reviews) Oct 14, 2026
@valdo404 valdo404 changed the title docs: explorations — 3D from imagery tiles, and imagery tile enhancement (literature reviews) docs: explorations — 3D from imagery, tile enhancement, neural textures, ML on WebGPU/wgpu (research notes) Oct 14, 2026
@valdo404
valdo404 force-pushed the claude/3d-tiles-shape-transformation-syp59p branch 3 times, most recently from 2b3a855 to 64daf6d Compare October 20, 2026 19:58
claude and others added 11 commits October 21, 2026 15:43
…tile meshes

Literature review (mesh segmentation, LoD2 reconstruction, procedural
inversion, shadow/albedo recovery), open reference datasets, a sketch
pipeline and the licensing constraint on commercial photorealistic tiles.
Nothing implemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3
Raster imagery + terrain tile -> plausible 3D scene (relief, buildings,
trees, structures), judged on credibility rather than accuracy. Covers
monocular height, off-nadir and shadow cues, satellite neural fields,
global height products, footprints and roof structure, LoD2, completion
trained with holes, facades, vegetation, shadow removal, satellite-to-city
generation, inverse procedural modeling and 3D foundation models; plus
reference datasets, licensing constraints, credibility metrics, a sketch
pipeline and experiments. Nothing implemented.

The earlier mesh-based note now points to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3
…nable in Tuile

Detail and super-resolution, cross-zoom consistency, hallucination
measurement, deblurring/MTF, denoising, JPEG artifact removal, all-in-one
restoration, white balance, exposure/tone (LUT and bilateral-grid fields),
colour transfer and multi-source harmonisation, colour science, haze,
atmospheric correction, cloud removal and compositing, seasons, transient
objects and inpainting, and quality assessment. Complements doc 18.

Opens with what is actionable: three render-side fixes in our own chain
(anisotropic sampling in the viewer, linear-light mips, reprojection
kernel), then offline enhancement stored as a derived layer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3
…ramids

Neural texture compression (NTC and its BC-latent descendants), filtering
of neural textures, the GPU substrate (cooperative vectors/matrices,
wgpu 29, WebGPU f16 and subgroups), learned image codecs and scalable
coding, implicit neural representations and overfitted codecs, 2D
Gaussian images, geospatial fields and embedding tiles. Judged against
tile constraints (borders, parent/child consistency, never black,
determinism, static view, RGB only, encode throughput), with tracks and
experiments for Tuile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3
webgpu-neural-networks.md: platform support (Linux the gap), JS runtimes
(ORT-Web, Transformers.js, LiteRT.js, WebLLM, WebNN), the Rust ecosystem
(Burn/CubeCL device sharing, versions checked on crates.io), compilers
emitting WGSL, interop, published trials by model class, dispatch overhead
as the bottleneck, limits and features, kernels, numerics, deployment, and
guidance and experiments for a Rust/wgpu engine.

onnx-webgpu-wgpu.md: ONNX essentials (opsets, contrib ops, external data,
quantization, export), ONNX Runtime Web and native WebGPU EP (Dawn),
Rust ONNX paths (burn-onnx, ort/ort-web, tract, RTen, candle, wonnx),
ONNX-to-WGSL compilers, pitfalls, and a recommendation per target.

Checked in published sources: wgpu 29.0.3 exposes no way to extract or
adopt a JS GPUDevice (backend module private), so JS runtimes cannot share
Tuile's device; cubecl-wgpu 0.10 is on wgpu 29, 0.11 on wgpu 30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3
…yers

A derived layer is a pure function of source tiles (tile, halo, parent,
other dates) and a versioned model, computed on first request, stored and
served like any layer. In tuile-tile-server the GPU worker is just another
Upstream; the client always shows the raw tile and swaps when the derived
one is resident. Covers the three tiers (browser, proxy, farm), naming,
determinism and provenance, licences (stored derivatives for open or own
sources only), a cost model to fill by measurement, and open questions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhevLiW3dUtveoxxmG9fg3
…deepest imagery level, measured on the pyramid's own (z, z+1) pairs
…eepest imagery level — dataset of damaged and cross-level pairs, before/after browser
@valdo404
valdo404 force-pushed the claude/3d-tiles-shape-transformation-syp59p branch from 64daf6d to 74a14b6 Compare October 21, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants