fix(packaging): include readme-renderer's dependency versions in renderer_version - #20516

Fixes #19232

Problem

update_description_html re-renders a description whenever readme-renderer's own version doesn't match the stored rendered_by. But readme-renderer delegates the actual rendering to docutils, Pygments, nh3, and a markdown backend (cmarkgfm historically, comrak in the version we currently pin). A bump to any of those can change the rendered HTML while readme-renderer's own version stays put, so the periodic task never notices and the stale descriptions are never re-rendered.

Fix

renderer_version() now returns a composite package==version string covering readme-renderer plus whichever of its rendering dependencies are installed, e.g.:

readme-renderer==45.0,docutils==0.23,Pygments==2.20.0,nh3==0.3.6,comrak==0.0.12

Dependencies that aren't installed are skipped, so listing both markdown backends is safe. The result is cached with functools.cache: the installed distribution set can't change within a process lifetime, and importlib.metadata.distribution() is an uncached sys.path walk that renderer_version() sits in front of on the upload path.

rendered_by is an unconstrained Text column and its only reader is the admin release-detail page, which prints it verbatim, so the longer, human-readable string is safe and keeps the value debuggable at a glance (which is why I kept it as a readable string rather than hashing the dependency set).

Operational note

Every stored rendered_by today is a bare version string, so none of them will match the new composite format and update_description_html will re-render every description once. That task is .limit(500) on a */5 crontab, i.e. 144,000 rows/day, so with PyPI's description count this is a multi-week pass rather than a spike. It isn't free, though, and it's worth deciding whether you want the batch limit raised or a one-off backfill instead.

This is the same event that already happens on every readme-renderer bump today, since the filter is a plain != on the whole value. What changes is the frequency: full invalidation moves from readme-renderer's release cadence to any dependabot bump of docutils/Pygments/nh3/comrak. If that's too often for the fleet to converge, happy to adjust.

Open question

I hardcoded the dependency list to the packages named in the issue. The alternative is deriving it from importlib.metadata.requires("readme-renderer") (which returns ['nh3>=0.2.14', 'docutils>=0.21.2', 'Pygments>=2.5.1', 'comrak>=0.0.11; extra == "md"']), so a future backend swap would be picked up automatically. I went with the explicit list because it keeps you in control of what can trigger a fleet-wide re-render, and it already covers both markdown backends. Happy to switch if you'd prefer the derived version.

Testing

Added unit tests for the composite format, for gracefully skipping a dependency that isn't installed, and for the actual regression: bumping only a dependency's version changes renderer_version(). Verified the new tests fail against the old implementation. tests/unit/utils/test_readme.py and tests/unit/packaging/test_tasks.py pass, warehouse/utils/readme.py is at 100% statement and branch coverage, and ruff check, ruff format --check, and mypy are clean.

Prepared with AI assistance (Claude).