TLD DNSSEC Timeline

Two parts:

1. A Python tool that, daily, fetches the DNS root zone, enumerates every delegated TLD (noting which carry a DS record, and with which algorithms), and queries each one's SOA through a validating resolver — recording whether the answer was DNSSEC-validated (AD bit), unauthenticated as expected for an unsigned TLD, or failed, with Extended DNS Error (EDE, RFC 8914) detail to tell DNSSEC failures apart from connectivity failures.
2. A static web app that renders a timeline of those measurements, with toggles for gTLD / ccTLD × IDN / non-IDN, and a per-day drill-down.

See PLAN.md for the design and decisions.

measure/        Python package (the measurement tool)
web/            static single-page app (timeline + drill-down)
data/           output: measurements/YYYY-MM-DD.json, index.json, timeline.json,
                idn-cctlds.json (generated IDN ccTLD mapping)
tools/          update_idn_cctlds.py, import_csv.py (future: legacy CSV -> JSON)
tests/          pytest suite + a root-zone fixture
.github/        CI (tests) and the daily measurement workflow

pip install .            # or:  pip install -e ".[dev]"  for tests

python -m measure --resolver 127.0.0.1 --output data

Useful flags: --port, --timeout, --concurrency, --date YYYY-MM-DD, --root-zone-file PATH (parse a local zone instead of fetching), --root-zone-url URL, --retries, and — see Silence is not a verdict below — --recheck-delay SECONDS (negative to skip the second pass) and --control-probes N.

Each run writes data/measurements/YYYY-MM-DD.json and rebuilds data/index.json and data/timeline.json from all daily files present, so the derived files stay consistent across re-runs and imports.

The raw ad, rcode, and full ede list are stored in every record, so the classification can be revisited without re-measuring.

A TLD that does not answer tells us nothing on its own: it may be unreachable, or we may have gone blind. Those are indistinguishable in a single observation, and conflating them is not hypothetical — every daily run from a hosted runner used to record a contiguous alphabetical band of 1–3 TLDs as unreachable while their authoritative servers were demonstrably healthy.

So a bare timeout is no status at all. It is recorded as unmeasured, which is deliberately not a member of STATUSES and never reaches the timeline; promoting it to a real status takes corroboration:

1. A deferred second pass (--recheck-delay, default 120s) re-observes every TLD that did not come back secure/insecure. In-pass --retries fire back-to-back and so only absorb packet loss; the delay is what makes the second observation independent of the first.
2. Two control probes, asked at the moment of each silence. Both bypass our own resolver and query authoritative servers directly — a cached answer proves nothing, and with RFC 8198 aggressive NSEC a resolver can synthesize an NXDOMAIN for a nonce name without emitting a packet:
   • vantage_ok — can we reach the root servers at all? Every one asked (--control-probes, default 3) must reply.
   • authorities_ok — do the TLD's own nameservers answer a direct query, at the glue addresses from the root zone? One reply is enough.

A silence becomes unreachable only if it survived both passes, we could demonstrably see, and the TLD's own authorities were themselves silent to a direct probe. Anything else stays unmeasured and is held out of results entirely, so the day shows an honest gap for that TLD (the same - the derived files already use for a TLD that was not delegated) rather than a status we never established. Held-out records are kept in the day file's unmeasured list, and every re-checked record carries a checks audit trail of both observations and their control verdicts.

Both probes are biased the same way: uncertainty costs us a data point rather than costing a TLD its reputation. Notably, a TLD with no addresses in the root zone can never be called unreachable — "we could not ask" is not evidence that "they did not answer".

Each TLD is tagged gTLD/ccTLD × IDN/non-IDN. Non-IDN names use the usual rule (two ASCII letters ⇒ ccTLD). IDN names (xn--…) can't be told apart by length, so the country-code ones come from a checked-in mapping, data/idn-cctlds.json. There is no hardcoded fallback: if the file is missing, every xn-- TLD classifies as gTLD until the mapping is generated.

tools/update_idn_cctlds.py regenerates the mapping from IANA's Root Zone Database:

python tools/update_idn_cctlds.py --output data/idn-cctlds.json

• It keeps xn-- TLDs whose Type is country-code.
• Sanity threshold: if the scrape yields fewer than 10 IDN ccTLDs or fewer than 10 IDN gTLDs, it treats the page as broken, leaves the existing file untouched, and exits non-zero (a "scraping failure").
• It only rewrites the file when the mapping actually changed, so the daily job doesn't make empty commits.

The daily workflow runs it before each measurement, commits a mapping change as its own commit (separate from the measurement), and reuses the previous mapping on a scraping failure. To still notify the repo owner, the workflow runs the measurement and publishes as normal, then fails the job at the very end if the scrape failed.

The tool trusts the resolver's AD bit and EDE; it does not validate itself. Run your own validating resolver for trustworthy results. The daily workflow installs Unbound and configures it minimally:

server:
  interface: 127.0.0.1
  auto-trust-anchor-file: "/var/lib/unbound/root.key"   # via unbound-anchor
  ede: yes            # EDE is off by default — enable it (Unbound >= 1.13.2)

Public EDE-capable resolvers (1.1.1.1, 8.8.8.8, 9.9.9.9) work as a cross-check via --resolver, but rate limits and third-party validation make your own resolver preferable.

A validating recursive resolver must reach authoritative servers on UDP/TCP port 53. GitHub-hosted runners generally allow this; if a runner blocks it, use a self-hosted runner.

Static files under web/, no build step. The timeline renders from timeline.json as a 100%-stacked area chart (share of TLDs by status):

• the stack is ordered with the most frequent status on top, least frequent at the bottom (near the axis, where the interesting failures live);
• the log scale absolute toggle switches from the default linear-percentage view to absolute TLD counts on a logarithmic y-axis, so rare statuses (e.g. bogus) stay visible despite secure dominating;
• clicking a legend entry hides that status; shares stay normalised against the day's full total, so a hidden status leaves a blank gap up to 100% rather than expanding the remaining bands;
• insecure starts hidden: it means the TLD is unsigned (or signed only with algorithms a validator may decline), which is a fact about DNSSEC adoption rather than about DNSSEC working, and it would otherwise add a large constant band to a chart about validation health;
• error starts hidden too, and the legend calls it measurement error: it collects the failures we could not attribute, which says more about the measurement than about the TLD;
• the four class toggles include/exclude gTLD/ccTLD × IDN/non-IDN;
• click-and-drag on the chart zooms into a date range (reset with the button); a plain click opens that day's detail;
• hovering shows absolute counts and percentages for every status, hidden ones included (dimmed), so a day's full composition stays readable even when bands are switched off; it stays readable with years of daily data.

Clicking a day fetches that day's file for the drill-down (a status "waffle" grid plus a searchable, sortable table). Shareable views: the URL hash (#YYYY-MM-DD) opens a day; ?scale=log, ?range=START,END, ?classes= and ?statuses= set the scale, zoom and toggles. Each is omitted when it matches the default, so ?statuses= appears as soon as the selection differs from "everything but insecure and error" — including when all statuses are shown.

Local preview (web files and data/ are siblings in the repo, so point the app at ../data/):

python -m http.server 8000        # from the repo root
# open http://localhost:8000/web/index.html?data=../data/

On the deployed site, index.html and data/ sit at the same level, so the default data/ path is used.

• .github/workflows/measure.yml runs daily: it installs Unbound, clones the gh-pages branch (which holds the accumulated history), runs the measurement into that tree so timeline.json aggregates all days, syncs the web/ files, and commits + pushes to gh-pages.
• One-time setup: in the repository settings, enable GitHub Pages with source = branch gh-pages, folder / (root). The first scheduled (or manually dispatched) run creates the branch.
• .github/workflows/ci.yml runs the test suite on push / PR.

pytest -q

tools/import_csv.py is a stub. It will transform the legacy CSV into the same daily-JSON schema (see the docstring there); the column mapping is defined once the CSV format is known.