Schema layer, CONTRIBUTING, error-report pipeline, disclaimer, CI, and re-checked city research - #12
Open
inkxel wants to merge 3 commits into
Open
Schema layer, CONTRIBUTING, error-report pipeline, disclaimer, CI, and re-checked city research#12inkxel wants to merge 3 commits into
inkxel wants to merge 3 commits into
Conversation
…ror-report pipeline, disclaimer, and re-checked city research Design layer (schema/): - common-schema.md: normalized per-feature shape, two required timestamps (data_as_of vs source.last_synced), untyped geometry, six-state per-category status enum, and jurisdiction-wide default rules (the Walnut, CA case) as a schema-level concept distinct from block/zone data. - coverage-registry.md: the new data/coverage_registry.json shape (per-category status + population), replacing the old flat status+array shape that couldn't represent a confirmed gap distinctly from "not built yet." - error-report-pipeline.md: the anti-spam design for GitHub-issue-based reports -- no backend, no anonymous auto-post, a real GitHub account to submit through IS the moderation step. Implementation: - Migrated data/coverage_registry.json and the three LA adapter scripts to the new schema (data_as_of set explicitly, not just source.last_synced). - app.js: isCovered() helper for the new per-category registry shape; shared confidenceBadge() across all three category panels (previously permits-only); reportIssueUrl()/reportLink() wiring a "Report a problem" link into every panel; first-run disclaimer modal (localStorage-gated) alongside the persistent footer strip. - .github/ISSUE_TEMPLATE/data-issue.yml + unsigned-rule.yml for the two report types the pipeline needs. - CONTRIBUTING.md: dashboard-tracing method, the common schema, the worked LA adapter example, and how to register a jurisdiction. Research: - Investigated ParkUsher directly (not open source, no API, meters paywalled, uneven coverage) -- doesn't change Seattle/SF prioritization. - Re-checked Chicago's meters gap (held up, with a real-but-unverified private fallback surfaced) and Seattle's sweeping gap (overturned -- real data on a second ArcGIS portal, with a source-confirmed voluntary-enforcement caveat) via the dashboard-tracing method. - Added San Diego as a researched adapter candidate. - Checked municipal-code hosting for LA/NYC/Chicago/Seattle/DC. - Updated SPEC.md's coverage table, ranked candidate list, and Next Steps checklist throughout to reflect all of the above. Known limitation: this session's network policy blocked direct HTTPS to arcgis.com/data.seattle.gov/data.sandiego.gov, so the Seattle and San Diego dashboard-traces are corroborated by search, not confirmed by live query -- flagged explicitly in both the affected research files and SPEC.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbWncp7hgMyXKDvX4oUcBM
Two of the five open items from the last round were achievable without network access; the other three (a second real adapter, live-verifying Seattle/San Diego, and the Walnut jurisdiction-rule entry) are re-confirmed still blocked on the same restrictive sandbox network policy -- checked directly again rather than assumed. - scripts/validate_schema.py: checks adapter GeoJSON output against schema/common-schema.md's required fields, and coverage_registry.json against schema/coverage-registry.md's status enum. Turns the schema docs into something enforced, not just documented. - .github/workflows/lint.yml: runs the validator (registry mode), Python syntax, YAML validity, and app.js syntax on every PR. - .github/workflows/label-data-issues.yml: parses new data-issue/unsigned-rule reports for their Jurisdiction/Category fields and applies city:*/category:* labels automatically, closing the manual-triage step schema/error-report-pipeline.md flagged as a future improvement. - CONTRIBUTING.md: documents the validator as a pre-PR step and the new CI. - SPEC.md: updated Next Steps to reflect what closed vs. what's still genuinely blocked, with the network check re-run and cited directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbWncp7hgMyXKDvX4oUcBM
Picked up PR #12 locally per its own handoff instructions: all four scripts/fetch_*.py adapters run clean from a local environment (LA + Census TIGERweb both reachable), and scripts/validate_schema.py confirms the real parking-data adapters + coverage registry validate against the common schema with zero violations. One new finding: validate_schema.py's `all` mode also checks data/national-places.geojson against the common schema, which is scoped to parking-rule categories only (sweeping/meters/permits/crime) per its own spec -- the boundary layer isn't one of those. Cosmetic (~59K false-positive violations in `all` only), not a data problem, not fixed yet. Chicago adapter is next but not started this round.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Works through essentially all of SPEC.md's open "Next steps" in two batches.
Design layer (new
schema/):common-schema.md— normalized per-feature shape: two required timestamps (data_as_ofvssource.last_synced, which diverge exactly when it matters — e.g. LA's frozen 2015 permit data), untyped geometry, a six-state per-category status enum (built/in_progress/gap/not_applicable/unconfirmed/paused), and jurisdiction-wide default rules (the Walnut, CA unsigned-rule case) as a schema-level concept.coverage-registry.md— the newdata/coverage_registry.jsonshape (per-category status + population), replacing the old flat shape that couldn't represent a confirmed gap distinctly from "not built yet."error-report-pipeline.md— the anti-spam design for the GitHub-issue-based report flow: no backend, no anonymous auto-post; a real GitHub account to submit through is the moderation step.Implementation:
data/coverage_registry.jsonand all three LA adapter scripts to the new schema.app.js—isCovered()for the new registry shape; a sharedconfidenceBadge()across all three category panels (previously permits-only);reportIssueUrl()/reportLink()wiring a "Report a problem" link into every panel; a first-run disclaimer modal (localStorage-gated) alongside the persistent footer..github/ISSUE_TEMPLATE/data-issue.yml+unsigned-rule.ymlfor the two report types.CONTRIBUTING.md— dashboard-tracing method, the schema, a worked LA-adapter example, how to register a jurisdiction.scripts/validate_schema.py— enforces the common schema and registry schema against real files, not just documents them..github/workflows/lint.yml— runs the validator, Python/YAML/JS syntax checks on every PR..github/workflows/label-data-issues.yml— auto-appliescity:*/category:*labels on new reports by parsing the issue-form body.Research:
Known limitation
This session's sandbox blocks direct HTTPS to
arcgis.com,data.seattle.gov,data.sandiego.gov,data.cityofchicago.org, and evenunpkg.comat the network-policy level (re-confirmed directly, not assumed). That leaves three things genuinely still blocked, flagged explicitly in SPEC.md rather than glossed over:scripts/) — writing one against guessed column names without being able to confirm them wasn't worth the risk of shipping something that looks done but silently isn't.place_id, and fabricating one would plant wrong data in the one file this project treats as hand-maintained ground truth.Test plan
python3 -m py_compile scripts/*.pynode --check app.jspython3 scripts/validate_schema.py all(registry + any adapter output present)confidenceBadge()date-threshold logic andreportIssueUrl()query-param construction verified in isolationlabel-data-issues.ymlverified against realistic form bodies🤖 Generated with Claude Code
https://claude.ai/code/session_01YbWncp7hgMyXKDvX4oUcBM
Generated by Claude Code