-
Notifications
You must be signed in to change notification settings - Fork 228
feat(3d-tiles): add traversal observability and benchmark baselines #4018
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
8179c72
feat(tiles): expose traversal work sets for diagnostics
ibgreen 429a34d
feat(tiles): add deterministic traversal observability snapshots
ibgreen c2e334f
feat(tiles): export traversal observability contract
ibgreen dc4b1cf
docs(3d-tiles): add observability and benchmark guide
ibgreen e404d73
docs(3d-tiles): link observability guide from concepts index
ibgreen cb4bd6e
docs(3d-tiles): link observability guide from module overview
ibgreen 18a11ca
docs(tiles): link traversal diagnostics from Tileset3D API
ibgreen 391f374
docs: note 3D Tiles observability tranche
ibgreen 5e33a84
test(tiles): cover deterministic traversal snapshots
ibgreen a6fa4ad
style(tiles): format observability export
ibgreen f61e962
style(tiles): format observability snapshot
ibgreen 11eb1e7
style(tiles): apply formatter output
ibgreen 7c05ee0
style(tiles): match biome type declaration formatting
ibgreen 4ecd5dd
fix(tiles): discard stale viewport snapshot state
ibgreen 5b7e4b6
docs(tiles): document implicit observability fields
ibgreen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
137 changes: 137 additions & 0 deletions
137
docs/modules/3d-tiles/concepts/observability-and-benchmarks.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| --- | ||
| title: Runtime observability and benchmark baselines | ||
| description: Make 3D Tiles traversal behavior measurable, reproducible, and performance-budgeted. | ||
| hide_title: true | ||
| page_style: designed | ||
| --- | ||
|
|
||
| import {Tiles3DDocsTabs} from '@site/src/components/docs/tiles-3d-docs-tabs'; | ||
| import {DocPageHeader} from '@site/src/components/docs/doc-page-header'; | ||
| import {DocOrientation, ReferenceBoundary} from '@site/src/components/docs/designed-doc'; | ||
|
|
||
| <DocPageHeader | ||
| eyebrow="3D Tiles runtime" | ||
| title="Measure traversal before tuning it." | ||
| description="Deterministic snapshots and source diagnostics make screen-space error, request pressure, and cache behavior comparable across cameras, devices, and releases." | ||
| tone="violet" | ||
| meta={['Deterministic snapshots', 'Request and cache counters', 'Benchmark budgets']} | ||
| /> | ||
|
|
||
| <Tiles3DDocsTabs active="runtime" /> | ||
|
|
||
| <DocOrientation | ||
| eyebrow="A small inspection contract" | ||
| title="Capture what the traverser decided." | ||
| description="The observability API reports selection and work sets without coupling loaders.gl to deck.gl, a renderer, or a particular benchmark harness." | ||
| tone="violet" | ||
| items={[ | ||
| {label: 'Selection', value: 'Sorted selected, requested, and empty tile IDs'}, | ||
| {label: 'Work', value: 'Loading, loaded, failed, and cached counts'}, | ||
| {label: 'Memory', value: 'Estimated resident bytes and active maximum SSE'}, | ||
| {label: 'Sources', value: 'Implicit subtree request and cache counters when available'} | ||
| ]} | ||
| /> | ||
|
|
||
| <ReferenceBoundary | ||
| title="Runtime observability" | ||
| description="Use snapshots for regression fixtures and counters for live dashboards. The values describe the completed traversal frame and never change traversal policy." | ||
| tone="violet" | ||
| /> | ||
|
|
||
| ## Snapshot API | ||
|
|
||
| getTileset3DTraversalSnapshot(tileset) returns a serializable | ||
| Tileset3DTraversalSnapshot with sorted IDs and numeric counters: | ||
|
|
||
| ~~~typescript | ||
| import { | ||
| getTileset3DTraversalSnapshot, | ||
| Tileset3D | ||
| } from '@loaders.gl/tiles'; | ||
|
|
||
| await tileset.selectTiles(viewport); | ||
| const snapshot = getTileset3DTraversalSnapshot(tileset); | ||
|
|
||
| console.log(snapshot.selectedTileIds); | ||
| console.log(snapshot.requestedTileIds); | ||
| console.log(snapshot.maximumScreenSpaceError); | ||
| ~~~ | ||
|
|
||
| The snapshot is intentionally defensive. It copies tile IDs, does not retain tile or content | ||
| objects, and can be JSON-serialized as a golden fixture. Capture it only after selectTiles resolves; | ||
| asynchronous loads can change the next frame's selected set. | ||
|
|
||
| | Field | Meaning | Unit | | ||
| | --- | --- | --- | | ||
| | frameNumber | Completed traversal frame | count | | ||
| | selectedTileIds | Tiles selected for rendering, sorted | IDs | | ||
| | requestedTileIds | Tiles queued for content loading, sorted | IDs | | ||
| | emptyTileIds | Hierarchy-only or empty tiles visited, sorted | IDs | | ||
| | visibleTileCount | Length of the selected set | count | | ||
| | renderableTileCount | Selected tiles with ready render content | count | | ||
| | loadingTileCount | Tile and subtree loads in flight | count | | ||
| | loadedTileCount | Cumulative successful tile loads | count | | ||
| | failedTileCount | Cumulative failed tile loads | count | | ||
| | cachedTileCount | Tiles currently retained in the cache | count | | ||
| | cacheBytes | Estimated resident content memory | bytes | | ||
| | maximumScreenSpaceError | Active memory-adjusted SSE threshold | logical/CSS pixels | | ||
|
|
||
| The optional implicitTiling object is supplied by Tiles3DSource. Its request, materialization, pending, | ||
| and parsed-cache counters are useful for diagnosing subtree fan-out and cache reuse. | ||
|
|
||
| ## Deterministic conformance fixtures | ||
|
|
||
| A conformance fixture should fix the tileset JSON, viewport, options, and initial cache state. | ||
| Compare sorted IDs and counts rather than object identity or request completion order. A practical | ||
| fixture captures: | ||
|
|
||
| 1. one root traversal with content unavailable; | ||
| 2. the same viewport after content resolves; | ||
| 3. a second traversal proving cache reuse; and | ||
| 4. a camera move that changes only the expected branch. | ||
|
|
||
| Keep network access hermetic by injecting a resolver or in-memory source. Put large hierarchies and | ||
| long camera paths in the slow suite. Do not use snapshots to bless a regression: explain why a | ||
| selected tile, request, or count changed. | ||
|
|
||
| ## Benchmark dimensions and budgets | ||
|
|
||
| Record at least: | ||
|
|
||
| - initialization time and first useful frame; | ||
| - traversal time per viewport; | ||
| - number of selected, requested, loaded, failed, and cached tiles; | ||
| - resident bytes and peak resident bytes; | ||
| - implicit subtree requests, cache hits, and materialized headers; | ||
| - number of frames required to reach the target SSE. | ||
|
|
||
| Use a fixed browser, viewport size, camera path, network fixture, and warm/cold-cache label. Suggested | ||
| starting budgets are deliberately relative: a correctness change should not increase cold-start | ||
| traversal time by more than 10%, request count by more than 5%, or resident bytes by more than 10% | ||
| for the same snapshot. Calibrate absolute limits to the dataset and CI hardware before enforcing | ||
| them. | ||
|
|
||
| ## Reading the counters | ||
|
|
||
| A high requestedTileIds count with a small selected set usually indicates aggressive refinement, | ||
| a low maximumScreenSpaceError, or a projection/culling mismatch. A growing loadingTileCount means | ||
| the source or decoder is the bottleneck; a growing failedTileCount points to transport, content, or | ||
| extension errors. High cachedTileCount and cacheBytes with repeated evictions suggest a budget that | ||
| is too small for the working set. | ||
|
|
||
| For implicit tiling, many requestedSubtrees with few cacheHits can indicate unstable URL resolution | ||
| or an undersized parsed-subtree cache. materializedTiles measures hierarchy work, not renderable GPU | ||
| content. | ||
|
|
||
| ## What snapshots do not promise | ||
|
|
||
| Snapshots are diagnostic, not a renderer contract. Tile IDs and counters are stable for a fixed | ||
| source, options, and traversal implementation; they are not guaranteed to match across different | ||
| tileset revisions. The API does not expose styling, GPU buffer formats, draw order, or network | ||
| timings. I3S sources provide the common tile counters, but the implicit-tiling section is specific | ||
| to 3D Tiles. | ||
|
|
||
| See [screen-space error and LOD](/docs/modules/3d-tiles/concepts/screen-space-error-and-lod), | ||
| [caching and memory](/docs/modules/3d-tiles/concepts/caching-and-memory), and | ||
| [request scheduling](/docs/modules/3d-tiles/concepts/request-scheduling-and-priorities) for the | ||
| controls that explain these measurements. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
100 changes: 100 additions & 0 deletions
100
modules/tiles/src/tileset-3d/common/tileset-observability.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| // SPDX-License-Identifier: MIT | ||
| // Copyright (c) vis.gl contributors | ||
|
|
||
| import type {Tileset3D} from './tileset-3d'; | ||
| import type {Tile3D} from './tile-3d'; | ||
|
|
||
| /** | ||
| * A deterministic, renderer-neutral snapshot of one {@link Tileset3D} traversal. | ||
| * | ||
| * Tile IDs are sorted so snapshots can be compared across runs even when child requests complete | ||
| * in a different order. Counts are reported for the completed traversal frame and cache values are | ||
| * expressed in bytes. This is an inspection contract; it does not alter traversal or request policy. | ||
| */ | ||
| export type Tileset3DTraversalSnapshot = { | ||
| /** Monotonically increasing traversal frame number. */ | ||
| frameNumber: number; | ||
| /** IDs selected for rendering, sorted for stable comparisons. */ | ||
| selectedTileIds: string[]; | ||
| /** IDs whose content was requested, sorted for stable comparisons. */ | ||
| requestedTileIds: string[]; | ||
| /** IDs visited as hierarchy-only or empty tiles, sorted for stable comparisons. */ | ||
| emptyTileIds: string[]; | ||
| /** Number of selected tiles in the completed frame. */ | ||
| visibleTileCount: number; | ||
| /** Number of selected tiles with renderable content. */ | ||
| renderableTileCount: number; | ||
| /** Number of tile or subtree loads currently in flight. */ | ||
| loadingTileCount: number; | ||
| /** Cumulative number of tiles loaded into the runtime cache. */ | ||
| loadedTileCount: number; | ||
| /** Cumulative number of failed tile loads. */ | ||
| failedTileCount: number; | ||
| /** Number of tiles currently retained in the runtime cache. */ | ||
| cachedTileCount: number; | ||
| /** Estimated cached content bytes. */ | ||
| cacheBytes: number; | ||
| /** Active memory-adjusted maximum SSE in logical/CSS pixels. */ | ||
| maximumScreenSpaceError: number; | ||
| /** Source-specific implicit subtree counters, when the source exposes them. */ | ||
| implicitTiling?: { | ||
| /** Number of subtree resources requested from the source. */ | ||
| requestedSubtrees: number; | ||
| /** Number of subtree resources successfully materialized. */ | ||
| loadedSubtrees: number; | ||
| /** Number of requests served by the parsed-subtree cache. */ | ||
| cacheHits: number; | ||
| /** Number of parsed subtrees currently retained for reuse. */ | ||
| cachedSubtrees: number; | ||
| /** Number of subtree requests currently in flight. */ | ||
| pendingSubtrees: number; | ||
|
ibgreen marked this conversation as resolved.
|
||
| /** Number of runtime tile headers created from materialized subtrees. */ | ||
| materializedTiles: number; | ||
| }; | ||
| }; | ||
|
|
||
| /** | ||
| * Creates a stable observability snapshot from public {@link Tileset3D} state. | ||
| * | ||
| * The helper intentionally reads only public runtime fields and source diagnostics. It is safe to | ||
| * call from an instrumentation loop and does not retain tile or content references. | ||
| * | ||
| * @param tileset - Runtime whose most recent traversal should be inspected. | ||
| * @returns A serializable snapshot suitable for logs, regression fixtures, and benchmark output. | ||
| */ | ||
| export function getTileset3DTraversalSnapshot(tileset: Tileset3D): Tileset3DTraversalSnapshot { | ||
| const selectedTiles = tileset.selectedTiles.slice(); | ||
| const requestedTiles = tileset.requestedTiles.slice(); | ||
| const emptyTiles = tileset.emptyTiles.slice(); | ||
|
|
||
| const getTileIds = (tiles: readonly Tile3D[]): string[] => | ||
| tiles.map(tile => String(tile.id)).sort(); | ||
|
|
||
| const getStatCount = (name: string): number => { | ||
| const count = tileset.stats.get(name).count; | ||
| return typeof count === 'number' && Number.isFinite(count) ? count : 0; | ||
| }; | ||
|
|
||
| const implicitTilingSource = tileset.source as Tileset3D['source'] & { | ||
| getImplicitTilingStats?: () => Tileset3DTraversalSnapshot['implicitTiling']; | ||
| }; | ||
| const implicitTiling = implicitTilingSource.getImplicitTilingStats?.(); | ||
|
|
||
| return { | ||
| frameNumber: tileset.frameNumber, | ||
| selectedTileIds: getTileIds(selectedTiles), | ||
| requestedTileIds: getTileIds(requestedTiles), | ||
| emptyTileIds: getTileIds(emptyTiles), | ||
| visibleTileCount: selectedTiles.length, | ||
| renderableTileCount: selectedTiles.filter( | ||
| tile => tile.contentAvailable && Boolean(tile.content) | ||
| ).length, | ||
| loadingTileCount: getStatCount('Tiles Loading'), | ||
| loadedTileCount: getStatCount('Tiles Loaded'), | ||
| failedTileCount: getStatCount('Failed Tile Loads'), | ||
| cachedTileCount: getStatCount('Tiles In Memory'), | ||
| cacheBytes: tileset.gpuMemoryUsageInBytes, | ||
| maximumScreenSpaceError: tileset.memoryAdjustedScreenSpaceError, | ||
| ...(implicitTiling ? {implicitTiling} : {}) | ||
| }; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| import {describe, expect, test} from 'vitest'; | ||
|
|
||
| import {getTileset3DTraversalSnapshot} from '../../src'; | ||
|
|
||
| function createTile(id: string, renderable = false): any { | ||
| return { | ||
| id, | ||
| contentAvailable: renderable, | ||
| content: renderable ? {} : null | ||
| }; | ||
| } | ||
|
|
||
| function createTileset(overrides: Record<string, unknown> = {}): any { | ||
| const values: Record<string, number> = { | ||
| 'Tiles Loading': 1, | ||
| 'Tiles Loaded': 4, | ||
| 'Failed Tile Loads': 2, | ||
| 'Tiles In Memory': 3 | ||
| }; | ||
| return { | ||
| frameNumber: 7, | ||
| selectedTiles: [createTile('tile-b', true), createTile('tile-a')], | ||
| requestedTiles: [createTile('tile-c'), createTile('tile-a')], | ||
| emptyTiles: [createTile('empty-b'), createTile('empty-a')], | ||
| stats: {get: (name: string) => ({count: values[name] ?? 0})}, | ||
| gpuMemoryUsageInBytes: 4096, | ||
| memoryAdjustedScreenSpaceError: 8, | ||
| source: {}, | ||
| ...overrides | ||
| }; | ||
| } | ||
|
|
||
| describe('getTileset3DTraversalSnapshot', () => { | ||
| test('sorts IDs and captures runtime counters', () => { | ||
| const snapshot = getTileset3DTraversalSnapshot(createTileset()); | ||
|
|
||
| expect(snapshot).toEqual({ | ||
| frameNumber: 7, | ||
| selectedTileIds: ['tile-a', 'tile-b'], | ||
| requestedTileIds: ['tile-a', 'tile-c'], | ||
| emptyTileIds: ['empty-a', 'empty-b'], | ||
| visibleTileCount: 2, | ||
| renderableTileCount: 1, | ||
| loadingTileCount: 1, | ||
| loadedTileCount: 4, | ||
| failedTileCount: 2, | ||
| cachedTileCount: 3, | ||
| cacheBytes: 4096, | ||
| maximumScreenSpaceError: 8 | ||
| }); | ||
| }); | ||
|
|
||
| test('copies IDs and includes implicit subtree diagnostics when available', () => { | ||
| const tileset = createTileset({ | ||
| source: { | ||
| getImplicitTilingStats: () => ({ | ||
| requestedSubtrees: 2, | ||
| loadedSubtrees: 1, | ||
| cacheHits: 3, | ||
| cachedSubtrees: 1, | ||
| pendingSubtrees: 0, | ||
| materializedTiles: 8 | ||
| }) | ||
| } | ||
| }); | ||
|
|
||
| const snapshot = getTileset3DTraversalSnapshot(tileset); | ||
| tileset.selectedTiles[0].id = 'mutated'; | ||
|
|
||
| expect(snapshot.selectedTileIds).toEqual(['tile-a', 'tile-b']); | ||
| expect(snapshot.implicitTiling?.cacheHits).toBe(3); | ||
| }); | ||
| }); |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.