ERP Newsletter Automation

Production automation for the weekly ESRC Education Research Programme newsletter.

The system monitors education-policy sources, ingests relevant articles, classifies each item into the newsletter's editorial sections, enriches articles with AI-assisted summaries and topic metadata, and presents the review workflow in a curator dashboard. It is a human-in-the-loop editorial tool: the model accelerates triage and sorting, while curators keep control of judgement, selection, edits, and publication.

Before automation, newsletter production involved roughly 7 hours per week of manual source scanning, deduplication, sectioning, and summary writing. This repo turns that work into a monitored pipeline plus a Streamlit review app.

Status: live production system. Daily weekday ingestion feeds a weekly curator cycle.

• Problem solved: reduced a roughly 7-hour weekly newsletter-production workflow into a monitored scraping, classification, enrichment, and curator-review pipeline.
• My role: designed and implemented the data pipeline, classifier service, Streamlit dashboard, monitoring jobs, deployment path, and governance documentation.
• Production status: live internal system; weekday ingestion feeds a weekly curator cycle, with human review before publication.
• Headline evidence: held-out macro F1 0.725, top-2 accuracy 0.879, production-feedback loop, drift/fairness monitoring, incident records, model card, and threat model.
• Start here: docs/decisions/model_card.md, docs/decisions/threat_model_and_security.md, docs/decisions/evaluation_findings.md, and docs/deployment/huggingface_spaces_setup.md.
• Next improvements: add API rate limiting, prompt-injection filtering for LLM enrichment, Supabase RLS/per-user dashboard identity, and deeper API/dashboard tests.

100+ configured sources
(custom web scrapers, RSS feeds, Google Alerts)
        |
        v
Scraping pipeline
        |
        v
Supabase / Postgres
articles, predictions, curator decisions, monitoring logs
        |
        +----------------------+
        |                      |
        v                      v
Classifier API            Claude enrichment
FastAPI + Docker          summaries, topic sentence,
Hugging Face Space        geography/topic tags
        |
        v
Streamlit curator dashboard
Triage -> Select Categories -> Newsletter Draft -> Sources
        |
        v
Curator finalises and publishes the newsletter

GitHub Actions orchestrate scraping, classification, enrichment sweeps, weekly reset, health checks, drift monitoring, fairness audit, and Supabase backups.

• Source ingestion: 101 active source entries in src/scraping/sources.yml, split across custom web scrapers, RSS feeds, and Google Alerts.
• Article normalisation: shared URL cleanup, text extraction, date handling, relevance filtering, deduplication, and Supabase upserts.
• Section classifier: sentence-transformer embeddings (sentence-transformers/all-MiniLM-L6-v2) with a scikit-learn classifier.
• Model serving: FastAPI Docker service deployed on a Hugging Face Space, exposing /health, /predict, and Prometheus /metrics.
• AI enrichment: LLM-generated summaries, topic sentences, and geography/topic tags. OpenAI is the primary provider on the CI runner (which cannot reliably reach Claude); Claude is a best-effort fallback. Summaries have a full fallback ladder down to extractive/placeholder (see docs/decisions/).
• Curator dashboard: Streamlit app for triage, categorisation, draft assembly, source feedback, and password-gated write actions.
• Monitoring: pipeline health checks, self-healing sweeps for missing classifications/summaries, drift logging, and a fairness audit.
• Decision records: production incidents, modelling choices, data-layer design, monitoring design, and security notes are documented in docs/decisions/.

The classifier is intentionally scoped to the six newsletter sections that can be learned from historical newsletter data. Two internal update sections are written manually and excluded from classification. It is an assistive ranking/triage layer, not an autonomous publishing system.

The held-out vs real-world gap (0.705 → 0.630 weighted) is analysed in docs/decisions/evaluation_findings.md.

Manual-only sections (never classified): Update from Programme, Update from PI.

Python · Supabase/Postgres · scikit-learn · sentence-transformers · FastAPI · Docker · Hugging Face Spaces · Streamlit · GitHub Actions · OpenAI (primary) + Anthropic Claude (fallback) enrichment · Prometheus metrics.

The live classifier API runs as a Docker-based Hugging Face Space. The root Dockerfile builds the serving image for src/serving/api.py, and the Space reads the YAML frontmatter at the top of this README (sdk: docker, app_port: 8000).

render.yaml remains in the repo as legacy infrastructure-as-code from an earlier Render deployment; Render is not the current production target. See docs/deployment/huggingface_spaces_setup.md.

Install dependencies (full pipeline, or API only):

pip install -r requirements.txt
pip install -r requirements-api.txt   # classifier API only

Create a local .env with the variables for the parts you want to run:

SUPABASE_URL=https://<project>.supabase.co
SUPABASE_SERVICE_KEY=<service-role-key>
SUPABASE_ANON_KEY=<anon-key>
# LLM enrichment — OpenAI is primary on the runner, Claude is a fallback
ENRICH_PROVIDER=openai
OPENAI_API_KEY=<openai-key>
ANTHROPIC_API_KEY=<anthropic-key>
CLASSIFIER_API_URL=https://<owner>-<space>.hf.space
CLASSIFIER_API_KEY=<hf-token-if-space-is-private>
# Optional: when the runner routes the Claude FALLBACK through the Space proxy
ANTHROPIC_BASE_URL=https://<owner>-<space>.hf.space
PROXY_TOKEN=<shared-secret-for-the-proxy>

For the Streamlit dashboard, also set a Streamlit secret:

CURATOR_PASSWORD = "<dashboard-password>"

# Serve the classifier API
uvicorn src.serving.api:app --reload --port 8000

# Run the curator dashboard
streamlit run dashboard/app.py

# Scrape a date window
python -m src.scraping.run --since 2026-05-12 --until 2026-05-15

# Weekly inference against Supabase
python src/pipeline.py --inference

# Classify an existing local pull without re-pulling
python src/pipeline.py --classify-only

# Dry-run a single source
python -m src.scraping.try_source --source dfe --since 2026-05-01 --save

pip install -r requirements-dev.txt
python -m pytest tests/ -q
python -m pip check
python -c "from fastapi.testclient import TestClient; from src.serving.api import app; c=TestClient(app); print(c.get('/health').json())"

The pytest suite is intentionally lightweight and network-free; it covers URL normalisation, scraper helpers, and the Anthropic proxy-token client wiring.

Apply the migrations in migrations/ when bootstrapping a new Supabase project. The data layer is documented in docs/decisions/data_layer_design.md.

The classifier is treated as decision support, not editorial authority.

• The production model excludes metadata features, because SHAP analysis showed metadata encouraged source-type shortcuts.
• It predicts only the six learnable editorial sections.
• Curators can override labels, reject articles, edit summaries, and choose final inclusion.
• Monitoring checks drift, low confidence, missing summaries/classifications, and class/source distribution shifts.
• Retraining criteria and lifecycle notes are in docs/decisions/model_lifecycle.md and the model card.

Start here for the engineering story:

• docs/decisions/model_card.md
• docs/decisions/data_layer_design.md
• docs/decisions/model_lifecycle.md
• docs/decisions/monitoring_redesign_2026_06_11.md
• docs/decisions/evaluation_findings.md
• docs/decisions/threat_model_and_security.md
• docs/deployment/huggingface_spaces_setup.md
• docs/deployment/deployment_challenges.md

See LICENSE.