Skip to content

Street View: Fix depth map parsing when top left pixel is not sky - #45

Open
jonfroehlich wants to merge 1 commit into
sk-zk:masterfrom
jonfroehlich:depth-header-offset-uint8
Open

jonfroehlich wants to merge 1 commit into
sk-zk:masterfrom
jonfroehlich:depth-header-offset-uint8

Conversation

@jonfroehlich

Copy link
Copy Markdown
Contributor

The offset field in the depth map header is one byte, but parse_header reads it with get_uint16, which also consumes the byte after it. That byte is the first entry of the plane index array, so the parsed offset comes out as 8 + 256 * first_plane_index.

It is only correct when the top left pixel of the depth map happens to be sky, whose plane index is 0. Otherwise the index array is read starting past its own beginning, and then runs off the end of the buffer:

IndexError: index 520 is out of bounds for axis 0 with size 132808

parse_panorama_message calls parse_depth unguarded, so the whole panorama fails to fetch when depth is requested, and it is deterministic, so retrying never helps.

This is visible without any data. Byte 0 of the header is the header size, and it is 8, so bytes 0 to 7 are the header and byte 8 is already payload. A two byte read at index 7 therefore extends one byte past the header the format declares. The fix is to read the field as the single byte it is.

Two panoramas that currently raise, if it is useful to confirm against live data:

from streetlevel import streetview
streetview.find_panorama_by_id("03muttxTplho1WIJtBH10A", download_depth=True)  # offset parses as 520
streetview.find_panorama_by_id("0aScRkRfaP9rCKKSKmOULA", download_depth=True)  # offset parses as 2312

520 is 8 + 256*2 and 2312 is 8 + 256*9, i.e. their top left pixels sit on plane 2 and plane 9. Across a large sample of panoramas, a little under 1% had a non-sky pixel in the top left corner, so this is uncommon but not rare.

Tests

depth.py had no test coverage, so nothing exercised parse_header and the field width was never checked. The new tests build a depth map from the documented layout rather than capturing one: one with sky in the top left, which parses either way, and one with a wall there, which is the minimum needed to make the misread visible. The synthetic map reproduces offset 520 exactly, matching the live failure above.

Before the fix, test_parse_with_nonzero_first_plane_index fails with the IndexError; after it, the full suite passes (37 tests).

The offset field in the depth map header is one byte, but parse_header reads
it with get_uint16, which also consumes the byte after it. That byte is the
first entry of the plane index array, so the parsed offset comes out as
8 + 256 * first_plane_index.

It is only correct when the top left pixel of the depth map happens to be sky,
whose plane index is 0. Otherwise the index array is read starting past its
own beginning and runs off the end of the buffer:

    IndexError: index 520 is out of bounds for axis 0 with size 132808

Since parse_panorama_message calls this unguarded, the panorama cannot be
fetched at all when depth is requested, and the failure is deterministic, so
retrying never helps.

The header declares its own length in byte 0, and that value is 8, so bytes 0
to 7 are the header and byte 8 is already payload. Reading two bytes at index
7 therefore extends one byte beyond the header. Read the field as the single
byte it is.

depth.py had no test coverage, so nothing exercised parse_header. The new
tests build a depth map from the documented layout: one with sky in the top
left, which passes either way, and one with a wall there, which is the minimum
needed to make the offset misread visible.
@sk-zk sk-zk added the gsv Google Street View label Aug 6, 2026
jonfroehlich added a commit to ProjectSidewalk/sidewalk-panorama-tools that referenced this pull request Aug 7, 2026
…ad as gone

RED half of a test-first pair - all 7 new tests fail on the current seam,
and the flagship one fails INSIDE streetlevel's parser, demonstrating the
bug being routed around. They pin:

- The seam survives a payload whose first index byte is nonzero (a modelled
  zenith: tunnels, soffits, parking structures - the class streetlevel's
  uint16 offset misread kills on every attempt, unfixed through 0.12.11,
  fix pending in sk-zk/streetlevel#45), returning the correct raster in
  streetlevel order plus the plane bundle.
- An unrecognized envelope (an error JSON, a quota page that parsed) raises
  DepthPayloadError - transient, retried - and is never misread as "pano
  gone", which would ledger 'unavailable' and permanently write off a pano
  Google still serves.
- _compute_depth_raster: the mirror fixture reconstructed in payload order,
  all-no-plane when the plane list is empty, and PARITY with streetlevel's
  own decode on a payload upstream can read - so swapping the raster source
  changes nothing for the ~99% of panos both decoders handle.
- _decode_depth_planes rejects out-of-range plane indices (streetlevel
  fails the same payload with a bare IndexError mid-decode) and zero-area
  rasters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jonfroehlich added a commit to ProjectSidewalk/sidewalk-panorama-tools that referenced this pull request Aug 7, 2026
)

GREEN half of the test-first pair (7 red on the previous commit, 170 pass
now). Decided with Jon after his upstream fix (sk-zk/streetlevel#45) was
confirmed unmerged and absent from 0.12.11, the latest release and the
floor of our pin.

The seam no longer calls parse_panorama_id_response: its unguarded depth
decode misreads the payload's uint8 offset byte as a uint16, so any pano
with a modelled zenith (nonzero first index byte - tunnels, overpass
soffits, parking structures; a bit under 1% in a large sample) raised on
every attempt, was classed transient, and re-requested forever without
resolving. Now the seam reads the response code, the three orientation
scalars (with streetlevel's exact conversions), and the payload path off
the raw msg, decodes planes with _decode_depth_planes, and derives the
raster with _compute_depth_raster - the artifact's documented
reconstruction identity running forward, vectorized, with parity against
streetlevel's own decode pinned in CI for the payloads both can read. The
raster is handed over in streetlevel's mirrored order on purpose, so the
writer's #58 un-mirror and every CI pin on stored orientation are
untouched.

What remains of streetlevel in the depth path is api.find_panorama_by_id -
the protobuf-URL builder and fetch - with the session passed through
unchanged, so the request on the wire is identical. The WHY and the
revisit condition are written at the seam and in the offset canary test:
when upstream's fix ships and the pin moves past it, the bypass becomes
unnecessary rather than wrong - simplify back, or continue dropping the
dependency (#56 discussion).

Hardening that fell out: an unrecognized envelope (error JSON, quota page)
now raises DepthPayloadError - transient - instead of parsing as "pano
gone" and ledgering 'unavailable' for a pano Google still serves; the
decoder rejects out-of-range plane indices (streetlevel fails the same
payload with a bare IndexError mid-decode) and zero-area rasters. The
writer's mask invariant stays: no longer a cross-parser check, it now
guards the seam-to-writer flip plumbing and any raster/index divergence a
future edit introduces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gsv Google Street View

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants