CERF Bundle Repository Framework

A small, dependency-free toolchain for running a public ROM bundle repository that the CE Runtime Foundation launcher (source) can download from.

You point it at a Cloudflare R2 bucket, keep your ROMs in a local folder tree, and the framework does the rest: it packs each device's files into deterministic archives, builds a single signed-by-hash manifest.json, and publishes both to R2 incrementally. The launcher then reads that one manifest to list every available device and fetch its ROMs.

Storage subsystem: Cloudflare R2 (S3-compatible API for writes, public custom domain or r2.dev URL for reads). It is the only backend.

What lives where

The dataset is never committed to git. data/ is the checkout; R2 is the remote.

• Python 3.9 or newer. No third-party packages — everything is stdlib (urllib, zipfile, hashlib, hmac), including the S3 v4 request signing.
• A Cloudflare account with R2 enabled.
• Disk space for the ROM tree plus its archives (an archive lives next to its source, so budget roughly 2× your ROM payload).

.r2_token is a plain KEY=value file at the repo root, git-ignored, read by every script. Copy the template:

copy .r2_token.example .r2_token     # Windows
cp .r2_token.example .r2_token       # POSIX

Any R2_* environment variable overrides the file.

In the Cloudflare dashboard → R2 → Create bucket. Then, on the bucket's Settings tab, expose it for reads with either:

• A custom domain (recommended) — e.g. downloads.example.com. Gives you clean URLs, cache control, and it is the only option that makes download analytics possible.
• The r2.dev subdomain — zero-setup, rate-limited, fine for testing.

R2 → API → Manage R2 API Tokens → Create API token, permission Object Read & Write, scoped to your bucket. Copy the Access Key ID and Secret Access Key — the secret is shown once.

# Public read URL. This is the custom domain or r2.dev URL from 1.1 —
# NOT the S3 API endpoint. Used by pull.py and by the launcher.
R2_PUBLIC_BASE_URL=https://downloads.example.com

# S3 API credentials from 1.2. Used by push.py.
R2_ACCOUNT_ID=0123456789abcdef0123456789abcdef
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET=cerf-bundles

Optional keys:

R2_PREFIX caveat. The prefix is applied to writes only. Public reads concatenate R2_PUBLIC_BASE_URL with the plain relative path, so when you set R2_PREFIX=cerf you must also point R2_PUBLIC_BASE_URL at https://downloads.example.com/cerf.

python push.py --check

This lists the bucket, does a signed write, a signed read, a public read through R2_PUBLIC_BASE_URL, and cleans up its probe object. A failing public_get while the signed probes pass means the bucket has no public access configured (step 1.1), or R2_PUBLIC_BASE_URL does not point at it.

If the bucket already holds a published dataset, one command reconstructs the entire working tree:

python pull.py

It downloads manifest.json, then for every bundle pulls rom.zip and each additional package archive, verifies size + sha256 against the manifest, unpacks them into data/roms/<bundle>/, and writes that bundle's cerf.json back out from the manifest. The result is a complete, buildable source tree.

pull.py is destructive. It deletes data/roms/ and data/manifest.json before downloading. Anything you have not pushed is gone. It is a checkout, not a merge.

Nothing to pull — just create the tree by hand. One directory per device:

data/
└── roms/
    ├── cerfos/
    │   ├── cerf.json          # required
    │   └── NK.bin
    └── jornada720/
        ├── cerf.json
        ├── jornada720.bin
        ├── jorn720_eeprom.bin
        └── jlime_cf.img

The directory name is the bundle id. It becomes the manifest's name and appears verbatim in every public URL, so keep it a plain, lowercase, URL-friendly name (jornada720, devemu_wm65, zune_pavo).

Put every file the device needs directly in its folder — ROM images, EEPROM dumps, bootloaders, CF card images, extra downloads. Then write cerf.json, which is what tells the framework which of those files to actually ship.

cerf.json is mandatory in every bundle folder and serves two unrelated audiences:

1. The framework reads a small, fixed set of fields to decide what gets packed and shipped. That contract is defined below and is the only part this repository cares about.
2. The CERF launcher reads everything else — device names, OS versions, board ids, screen geometry, feature flags. The framework never looks at those keys, never validates them, and never rewrites them; it copies them into manifest.json verbatim, key order preserved, so the launcher can read a device's metadata without downloading a single byte of ROM.

Launcher-side metadata is out of scope for this framework. No schema for it is published on the CERF side yet; match what the launcher version you target expects. Any key you add reaches the launcher untouched.

The minimum the framework needs is one file selection:

{
  "rom": {
    "primary": "NK.bin"
  }
}

Nothing is packed unless you name it. rom.zip contains exactly the files listed in the rom block. A file in the bundle folder that no field names is neither packed nor published, so working copies can sit in the bundle folder without shipping.

Rules the build enforces (each is a hard error, not a warning):

• rom.primary must be present.
• Every named file must exist in the bundle folder.
• Package sources must be plain names — no /, no .., no subpaths.
• Each package source may be declared once across all categories.
• A file package must not also appear in the rom block. The two are different delivery channels: rom.zip is the device's ROM, packages are optional side downloads.
• A package source may not be named rom (it would collide with rom.zip).

Pass-through behavior:

• Category names are yours. The framework treats additional_packages keys as opaque labels and passes them through — compact_flash_cards, Files, demos, whatever the launcher wants to group by.
• Extra keys on a package entry pass through too, which is how a human-readable "name" reaches the launcher.

data/roms/jornada720/cerf.json — one primary ROM, an EEPROM dump, and a CF card image offered as a separate download:

{
  // ---- launcher metadata: forwarded verbatim ----
  "meta": {
    "device_name": "HP Jornada 720",
    "os": { "name": "Handheld PC 2000", "ver_major": 3, "ver_minor": 0, "year": 2000 },
    "device_year": 2000,
    "notes": ["JLime Linux: username: root, then do 'startx'"]
  },
  "board": {
    "id": "jornada_720",
    "configurable_screen_width": 640,
    "configurable_screen_height": 240
  },

  // ---- framework contract: selects what ships ----
  "rom": {
    "primary": "jornada720.bin",
    "eeprom": "jorn720_eeprom.bin"
  },
  "additional_packages": {
    "compact_flash_cards": [
      { "file": "jlime_cf.img", "name": "JLime Linux" }
    ]
  }
}

That bundle ships two archives: rom.zip (the two ROM binaries) and jlime_cf.img.zip (the card image). cerf.json is in neither — it travels inside manifest.json, so a metadata edit never re-uploads a ROM.

python make_bundles.py

For each bundle this writes, next to the sources:

• rom.zip — the binaries named in the rom block, at the archive root.
• <source>.zip — one per additional package.

and then regenerates data/manifest.json from scratch.

Archives are deterministic: members sorted, timestamps pinned to 1980-01-01, fixed permissions. Identical inputs produce an identical sha256, which is what lets the publish step skip unchanged archives.

The build is incremental. Each bundle keeps a .build_cache.json sidecar recording every archive's source signature (name, size, mtime). An archive is re-zipped only when that signature changes, so an untouched 2 GB ROM is reused instead of re-deflated — which matters because cerf.json changes far more often than ROMs do, and it lives outside every archive.

Stale archives are cleaned up as you go: rename or drop a package and the orphaned local .zip is deleted, along with its cache entry.

python push.py

The manifest is the source of truth. push.py uploads every archive the manifest references, then manifest.json last — so the published manifest never points at an object that is not there yet.

It is incremental too: it fetches the currently published manifest.json over the S3 API (strongly consistent, not CDN-cached) and skips any archive whose archive_sha256 already matches and whose object is still present. manifest.json is always re-uploaded, since it carries cerf.json.

Finally it deletes stale objects — anything under roms/ or the manifest itself that the current manifest no longer references. Delete a bundle locally, push, and it disappears from R2. Objects outside that managed scope (including analytics.json) are never touched.

Object size ceiling. Each archive is uploaded in a single PUT and buffered in memory: an archive must stay under R2's ~5 GiB single-request limit, and the push host needs RAM for the largest one. Split oversized payloads across several packages.

python orchestrate.py

Runs make_bundles.py then push.py, stopping at the first failure.

Public URLs, relative to R2_PUBLIC_BASE_URL:

/manifest.json
/roms/<bundle>/rom.zip
/roms/<bundle>/<source>.zip
/analytics.json                  (optional, see below)

manifest.json is the launcher's single entry point — version 2, bundles sorted by name:

{
  "version": 2,
  "abuse_email": "my@email.com",
  "bundles": [
    {
      "name": "jornada720",
      "updated_at": "2026-07-23T15:55:55Z",
      "archive_path": "roms/jornada720/rom.zip",
      "archive_sha256": "40dd80fa8aec6b62c635ea2d11a1f0f3baa8f9e46686159f74b20b29579fc7ae",
      "archive_size": 18409759,
      "unpacked_size": 33554688,
      "additional_packages": {
        "compact_flash_cards": [
          {
            "file": "jlime_cf.img",
            "name": "JLime Linux",
            "archive_path": "roms/jornada720/jlime_cf.img.zip",
            "archive_sha256": "cc91d1c0be55f6c5da0c3eaf615a9b341c4bdd031370ce663052f8c928f2e82d",
            "archive_size": 67296756,
            "unpacked_size": 528482304
          }
        ]
      },
      "cerf_json": { "…": "your cerf.json, verbatim" }
    }
  ]
}

Per entry: archive_* describes rom.zip; unpacked_size is the total uncompressed payload, so a client can check free space before downloading; additional_packages mirrors the cerf.json structure with each entry enriched by the same archive fields, or is null when the bundle declares none; updated_at advances only when the archives, the packages or cerf.json change; cerf_json is the bundle's whole cerf.json.

analytics.py builds a "most-downloaded ROMs" ranking from real traffic and publishes it alongside the dataset. It needs the bucket to be served through a Cloudflare custom domain — that is the only configuration where per-URL download counts exist.

Add to .r2_token:

# Cloudflare API token scoped to Zone → Analytics → Read for the download zone,
# plus that zone's id (zone Overview page, right sidebar).
CF_API_TOKEN=
CF_ZONE_ID=

Two phases per run:

1. Private collection. Reads per-rom.zip request counts (HTTP 200/206) from the Cloudflare GraphQL Analytics API. Cloudflare retains that data for only ~8 days, so each run backfills the last week into a local analytics_db.json (git-ignored, one small record per UTC day). That DB is what makes windows longer than Cloudflare's retention possible, and a missed run self-heals on the next backfill.
2. Public publication. Ranks bundles over a trailing window and uploads analytics.json next to manifest.json, using the same R2 credentials as push.py. The published format:

{ "per_rom": [ { "name": "jornada720", "place": 1 }, { "name": "cerfos", "place": 2 } ] }

python analytics.py                 # collect + publish — the cron command
python analytics.py --dry-run       # fetch, rank, print; write and upload nothing
python analytics.py --collect-only  # update the local DB only
python analytics.py --check         # validate Cloudflare + R2 credentials
python analytics.py --window-days 30 --fetch-days 7

Run it on a daily cron. Only downloads through the public custom domain are counted; objects fetched directly over the S3 API are invisible to this dataset.

.gitignore keeps data/, .r2_token, .r2.env, analytics_db.json and __pycache__/ out of history — never commit ROM payloads or credentials. .gitattributes pins text files to LF and marks ROM and archive extensions binary so git never mangles them.