This guide is for people who maintain the Index of Data Models — the data pipeline, the GitHub Actions workflows, and the site itself. If you just want to browse or use the schemas, start with the README and the live site.

• How it fits together
• The data pipeline
• GitHub Actions workflows
• Required secrets
• Running the pipeline locally
• Common tasks
• Conventions

There are three moving parts:

1. A React site (src/, built with Vite) — the browsable UI, deployed to GitHub Pages.
2. Generated data (public/data.json, public/staging_checks.json) — the schema list and staging results the site reads at runtime.
3. A Python pipeline (scripts/) — pulls from Synapse, validates, and publishes to CoreModels.

The site is a static build: at runtime it just fetches the two JSON files from public/. All the "intelligence" (what schemas exist, are they valid) is baked into those files by the pipeline. See README → How the data stays up to date for the flow diagram.

All three scripts live in scripts/ and are configured entirely through environment variables.

Lists every organization and schema in the Synapse JSON Schema registry and writes one row per schema (its latest version) to public/data.json. It caches by SHA-256 so unchanged schemas skip a redundant detail fetch.

The PUBLISHED_ORGS set near the top of the script decides which organizations are marked published (vs draft). See Common tasks.

Reads public/data.json, and for each published schema verifies its URI resolves in the Synapse staging registry. Optionally creates (and cleans up) a curation task per URI as a deeper check.

Reads public/data.json and imports each published schema into CoreModels via its Merge JSON Schema API.

(Additional tuning vars — VERIFY_URLS, POLL_INTERVAL_SECONDS, JOB_TIMEOUT_SECONDS, override flags — are documented in the script's docstring.)

Runs on a schedule (06:00 UTC daily), on manual dispatch (with an optional org_name filter), and on push to main. Jobs:

Because branch protection blocks direct pushes to main, automated data changes always arrive as a reviewable pull request. Merging that PR is what actually updates the public site and CoreModels.

Manual-only. Runs the same staging check against the develop build of the Synapse Python client and uploads the results as a workflow artifact. It never commits, deploys, or publishes — it's an early-warning smoke test for upcoming client releases.

Set these in Settings → Secrets and variables → Actions:

Create a .env file in the repository root (it is git-ignored) with the variables above, then run from the repo root so the default public/… paths resolve:

pip install synapseclient requests python-dotenv

# Regenerate the schema list (tip: scope to one org while testing)
ORG_NAME_FILTER=sage.schemas.dpe python scripts/export_orgs.py

# Validate against staging
python scripts/check_schema_uris_staging.py

# Preview a CoreModels import without writing anything
DRY_RUN=true python scripts/import_to_coremodels.py

To avoid overwriting the committed public/data.json while experimenting, point the scripts at a throwaway path with OUTPUT_PATH / DATA_JSON_PATH.

Edit the PUBLISHED_ORGS set in scripts/export_orgs.py and add the organization's exact organization_name. Published orgs are the ones shown by default, validated against staging, and imported into CoreModels.

Edit src/data/dcc.js — each entry has repo, schemas, docs, and portal links (use null for any that don't exist). Update the count in the TABS array in src/App.jsx if you change how many DCCs are listed.

Edit the cron expression under on.schedule in .github/workflows/update-data.yml.

Trigger Update Schema Data → Deploy Pages → Publish to CoreModels from the Actions tab (workflow_dispatch), optionally with an org_name filter. It will open a PR with any changes.

• Never hand-edit public/data.json or public/staging_checks.json — they're generated.
• Never commit secrets. Local credentials go in the git-ignored root .env; CI uses GitHub Secrets.
• Keep public/ for served assets only. Tooling and one-off scripts belong in scripts/, so they aren't published to the GitHub Pages site.
• The site's base path is /core-models/ (vite.config.js) because GitHub Pages serves it from a subpath. The app fetches data with relative URLs (./data.json), so it keeps working under that base.