Repository navigation
Street View: Fix depth map parsing when top left pixel is not sky - #45
Open
jonfroehlich wants to merge 1 commit into
Open
jonfroehlich wants to merge 1 commit into
jonfroehlich wants to merge 1 commit into
Conversation
The offset field in the depth map header is one byte, but parse_header reads
it with get_uint16, which also consumes the byte after it. That byte is the
first entry of the plane index array, so the parsed offset comes out as
8 + 256 * first_plane_index.
It is only correct when the top left pixel of the depth map happens to be sky,
whose plane index is 0. Otherwise the index array is read starting past its
own beginning and runs off the end of the buffer:
IndexError: index 520 is out of bounds for axis 0 with size 132808
Since parse_panorama_message calls this unguarded, the panorama cannot be
fetched at all when depth is requested, and the failure is deterministic, so
retrying never helps.
The header declares its own length in byte 0, and that value is 8, so bytes 0
to 7 are the header and byte 8 is already payload. Reading two bytes at index
7 therefore extends one byte beyond the header. Read the field as the single
byte it is.
depth.py had no test coverage, so nothing exercised parse_header. The new
tests build a depth map from the documented layout: one with sky in the top
left, which passes either way, and one with a wall there, which is the minimum
needed to make the offset misread visible.
jonfroehlich
added a commit
to ProjectSidewalk/sidewalk-panorama-tools
that referenced
this pull request
Aug 7, 2026
…ad as gone RED half of a test-first pair - all 7 new tests fail on the current seam, and the flagship one fails INSIDE streetlevel's parser, demonstrating the bug being routed around. They pin: - The seam survives a payload whose first index byte is nonzero (a modelled zenith: tunnels, soffits, parking structures - the class streetlevel's uint16 offset misread kills on every attempt, unfixed through 0.12.11, fix pending in sk-zk/streetlevel#45), returning the correct raster in streetlevel order plus the plane bundle. - An unrecognized envelope (an error JSON, a quota page that parsed) raises DepthPayloadError - transient, retried - and is never misread as "pano gone", which would ledger 'unavailable' and permanently write off a pano Google still serves. - _compute_depth_raster: the mirror fixture reconstructed in payload order, all-no-plane when the plane list is empty, and PARITY with streetlevel's own decode on a payload upstream can read - so swapping the raster source changes nothing for the ~99% of panos both decoders handle. - _decode_depth_planes rejects out-of-range plane indices (streetlevel fails the same payload with a bare IndexError mid-decode) and zero-area rasters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jonfroehlich
added a commit
to ProjectSidewalk/sidewalk-panorama-tools
that referenced
this pull request
Aug 7, 2026
) GREEN half of the test-first pair (7 red on the previous commit, 170 pass now). Decided with Jon after his upstream fix (sk-zk/streetlevel#45) was confirmed unmerged and absent from 0.12.11, the latest release and the floor of our pin. The seam no longer calls parse_panorama_id_response: its unguarded depth decode misreads the payload's uint8 offset byte as a uint16, so any pano with a modelled zenith (nonzero first index byte - tunnels, overpass soffits, parking structures; a bit under 1% in a large sample) raised on every attempt, was classed transient, and re-requested forever without resolving. Now the seam reads the response code, the three orientation scalars (with streetlevel's exact conversions), and the payload path off the raw msg, decodes planes with _decode_depth_planes, and derives the raster with _compute_depth_raster - the artifact's documented reconstruction identity running forward, vectorized, with parity against streetlevel's own decode pinned in CI for the payloads both can read. The raster is handed over in streetlevel's mirrored order on purpose, so the writer's #58 un-mirror and every CI pin on stored orientation are untouched. What remains of streetlevel in the depth path is api.find_panorama_by_id - the protobuf-URL builder and fetch - with the session passed through unchanged, so the request on the wire is identical. The WHY and the revisit condition are written at the seam and in the offset canary test: when upstream's fix ships and the pin moves past it, the bypass becomes unnecessary rather than wrong - simplify back, or continue dropping the dependency (#56 discussion). Hardening that fell out: an unrecognized envelope (error JSON, quota page) now raises DepthPayloadError - transient - instead of parsing as "pano gone" and ledgering 'unavailable' for a pano Google still serves; the decoder rejects out-of-range plane indices (streetlevel fails the same payload with a bare IndexError mid-decode) and zero-area rasters. The writer's mask invariant stays: no longer a cross-parser check, it now guards the seam-to-writer flip plumbing and any raster/index divergence a future edit introduces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
offsetfield in the depth map header is one byte, butparse_headerreads it withget_uint16, which also consumes the byte after it. That byte is the first entry of the plane index array, so the parsed offset comes out as8 + 256 * first_plane_index.It is only correct when the top left pixel of the depth map happens to be sky, whose plane index is 0. Otherwise the index array is read starting past its own beginning, and then runs off the end of the buffer:
parse_panorama_messagecallsparse_depthunguarded, so the whole panorama fails to fetch when depth is requested, and it is deterministic, so retrying never helps.This is visible without any data. Byte 0 of the header is the header size, and it is 8, so bytes 0 to 7 are the header and byte 8 is already payload. A two byte read at index 7 therefore extends one byte past the header the format declares. The fix is to read the field as the single byte it is.
Two panoramas that currently raise, if it is useful to confirm against live data:
520 is
8 + 256*2and 2312 is8 + 256*9, i.e. their top left pixels sit on plane 2 and plane 9. Across a large sample of panoramas, a little under 1% had a non-sky pixel in the top left corner, so this is uncommon but not rare.Tests
depth.pyhad no test coverage, so nothing exercisedparse_headerand the field width was never checked. The new tests build a depth map from the documented layout rather than capturing one: one with sky in the top left, which parses either way, and one with a wall there, which is the minimum needed to make the misread visible. The synthetic map reproduces offset 520 exactly, matching the live failure above.Before the fix,
test_parse_with_nonzero_first_plane_indexfails with theIndexError; after it, the full suite passes (37 tests).