clinic-triage

Stage 2 of the taildrop → sardinetracker chain. Decides which PDFs in the Taildrop inbox are clinical records, and proposes a description for each one.

┌─ T14 ───────────────────────────────────────────────────────┐
│  ~/Downloads/taildrop/*.pdf                                 │
│         │                                                   │
│         ├─ GATE 1  sha256 → manifest.json   seen it before? │
│         ├─ GATE 2  weighted classifier      is it clinical? │
│         ├─         pdftotext -layout + regex   date/provider│
│         └─ GATE 3  qwen-local via Ollama    title + summary │
│                        │                                    │
│                        ▼                                    │
│                   queue/<sha>.json   ← you read this        │
│                        │                                    │
│  push.py               ├─ GATE 4  you approve (y/n/edit)    │
│                        ▼                                    │
│              POST /api/clinical/document  (Bearer)          │
└────────────────────────┼────────────────────────────────────┘
                         ▼
        your-server.example.com/clinical#documents

triage.py never uploads anything — it only writes to queue/ and state/. push.py is the only thing here that talks to the server, and it sends nothing without a y.

A companion to sardinetracker, not a part of it. It is useful on its own if you already run the pieces it leans on: a machine that receives your own scanned records, a local model to describe them, and somewhere to file them. Swap the push.py target and the first three gates work against any document store.

Nothing here leaves your machine except the final POST, and that one waits for a y. The classifier, the date and provider regexes, and the model prompt all run locally; no document text is sent anywhere to decide what it is.

Your records are health data. state/, queue/, filed/, and your real config.json hold document text, clinician names, and an API token. They are gitignored for that reason — if you fork this, keep them that way.

Hashing is free, so it goes first: a re-sent file costs nothing at all. The classifier runs before the model because the model is the expensive step — around 90 seconds a document on this laptop — and because the plant's incident tickets should never be fed to a language model in the first place. By the time qwen sees anything, the document has already been established as clinical.

The first version scored clinical keywords against non-clinical ones. Against two documents it looked excellent. Against the 69 that actually arrived it passed 62 of 63 — including journal articles about lupus, legal drafts, patient-education leaflets, this tracker's own PDF exports, and a file whose entire contents were an account recovery code.

That failure is informative rather than embarrassing. The inbox is full of clinical vocabulary. A paper on anti-dsDNA isotypes says "diagnosis" and "laboratory" far more often than a visit summary does. Vocabulary cannot separate these; structure can.

So the test is now: does this document have the shape of a record generated by a provider about a specific patient?

Anchors — any one of these shapes qualifies it:

A demographics block (patient_block and dob_field) or a strong encounter structure (encounter ≥ 3) makes it clinical. Partial structure makes it ambiguous — described anyway, flagged for a read.

Vetoes — a document is rejected outright if it trips one. Thresholds come from measuring the real corpus, because genuine records do use these words occasionally; journal articles ran 6–80 hits where a visit summary saying "abstract" once ran 1.

A length floor (MIN_CHARS = 400) sits in front of everything. It exists because of one specific file: a PDF containing nothing but an account recovery code, 25 characters long. No content signal can catch that — there is no content. Anything below the floor is too_short, never described, never sent.

On the real inbox this yields 36 clinical, 2 ambiguous, 22 not clinical, 2 too short, 1 needing OCR. Every verdict prints its reason:

  - README.pdf  [not_clinical]
      not_clinical: looks like own report
  - Drug levels how to use 2026 Thomas Garg and Balevic.pdf  [not_clinical]
      not_clinical: looks like journal article

Run --dry-run -v to see the anchor and veto counts behind any single call.

The classifier answers "does this have the shape of a clinical record?". It cannot answer "do I want my clinicians reading this?", which is a different question and not one a heuristic should decide. Three mechanisms cover the gap.

An exclude list in config.json — filename globs, checked before anything else, before hashing or the model. Explicit, auditable, and yours:

"exclude": ["*complaint*", "*case_brief*", "*draft*"]

A self_authored veto. Documents the patient wrote score as clinical records — they are about clinical findings and use the same vocabulary. One was typed by the model as a clinic note with a summary in a clinician's voice, which is the version of this mistake that survives review. The signal is patient-voice phrasing (I have a history, my symptoms, I would like to request), deliberately not first person as such: physicians write "I examined", "I recommend" constantly. Across 69 documents it fires on one and touches none of the clinic notes.

A fileable-type allowlist. other stays among the types offered to the model so it can say "I don't recognise this" rather than guess — and that answer holds the document back instead of counting as a type. Blank intake paperwork lands here.

The exclude list deletes outright, being an explicit instruction. The other two write a proposal marked not_clinical or not_a_record, so a withheld document is visible and reversible rather than silently absent.

--refresh-roster pulls the roster from GET /api/clinicians and caches it in state/. Matching a known roster is deterministic where reading a name off a letterhead is not, and clinic_name fills the facility field that the letterhead heuristic rarely finds.

Two rules keep it honest:

Position. A document's own clinician appears in the header, the letterhead or the signature block. One merely referenced appears in the body ("follow up with Dr Example"). Matching anywhere attributed both ER visits, an urgent care note and a telehealth visit to the primary care doctor, because each says to follow up with him.

Referenced-role redaction. Portal exports open with a patient banner — Name: ... | DOB: ... | MRN: ... | PCP: A. Physician, MD — naming the patient's primary care doctor on page one of every document from that system. PCP:, Primary Care Provider:, Referring:, pharmacy and emergency-contact fields are blanked before any matching happens.

A match on the filename overrides the regex, because these files are named for the clinician seen: a cardiology consult was being attributed to the PCP named in its banner until the filename won.

Surnames that are ordinary words (Standard, Grant, Young) only match when anchored to a title, credentials, or the filename — otherwise "standard of care" is a match.

Not cosmetic. Without it the two-column header linearises and the labels separate from their values — Provider: ends up four lines above A. Physician MD, and a Provider:\s*(.+) regex returns nothing. With -layout the pair survives on one line.

Note that sardinetracker's own _extract_pdf_text() uses plain pdftotext without -layout. That is the right call there, because it feeds full-text search where layout only adds noise. This tool needs the opposite thing, so it does its own extraction rather than sharing that function.

Anything readable off the page is read off the page. Every proposal records a field_sources map, so a wrong field can be traced to the thing that produced it rather than blamed on "the AI".

The first version returned 1980-01-02 — a date of birth — as the date of service for six documents. Each fix came from a real failure:

1. The label and the value are on different lines. A visit note puts DOB: in a header row and 01/02/1980 in the row beneath it, so skipping "lines that mention DOB" misses the birthday entirely. The fix looks ahead.
2. The header row also carries the document's own date. That same row reads Visit Note - March 4, 2026 … DOB: … MRN:, so harvesting every date near the label blacklists the visit date. Only dates after the label count.
3. A lab header packs several fields onto one line. DOB: 01/02/1980 Age: 36 Specimen: … Collected: 03/04/2026 — taking every date after the label also blacklists the collection date, leaving the document with no date at all. Only the first date after the label is the birthday.

There is deliberately no "earliest date in the document" fallback. It was the mechanism by which all three failures produced a confident wrong answer. A blank date is safe; a wrong one is not.

The filename is the second source, and in this inbox it is a good one — these files are named by date of service. But it is checked: if the filename's date appears nowhere in the document, the source reads filename (NOT corroborated in the document). That caught a genuine problem — ER-2025-03-04.pdf and 2026-03-04-ER-pleurisy.pdf hold identical text whose own dates read 03/4/2026, so one filename is wrong by a year.

Matching "everything after the word Provider" harvested Diagnosis, Review, | Legal Name: Firstname, • Treatment and for further fills if inhalers help. A provider must now look like a name — capitalised words, optionally with credentials — and a short stoplist rejects form-field labels like First Name and Other Health Professional. A facility must look like an organisation, not a sentence or a heading. Both return None rather than a plausible fragment.

triage.py takes a lock in state/triage.lock. Two concurrent runs share a manifest and a queue directory, and the second silently overwrites the first — including replacing finished descriptions with un-described ones. If a run dies badly the lock may outlive it; the message names the pid so you can check.

./triage.py --reextract

Recomputes date, provider and facility for everything already in the queue and leaves the model's fields untouched. Use it after changing an extractor — the expensive half of the work is already paid for.

./install.sh                        # symlinks clinic-triage onto PATH and its man page
cp config.json.example config.json  # then fill in server, api_token, user_id
man clinic-triage                   # steps, commands, troubleshooting
clinic-triage doctor                # poppler, tailscale, ollama, inbox, server

install.sh symlinks rather than copies, so a git pull updates both the command and its man page. config.json is gitignored — it holds your token and the patient names the describer scrubs out of model output.

clinic-triage wraps the two scripts so the pipeline can be driven from anywhere. It echoes the underlying command before running it, so it stays a convenience rather than something to reverse-engineer later.

clinic-triage run     # collect from Taildrop, then describe what is new
clinic-triage push    # review each proposal and file the approved ones

Everything else — get, check, describe, preview, status, roster, repair, doctor — is in the man page.

cp config.json.example config.json     # server, api_token, user_id
./push.py --dry-run                    # show exactly what would be sent
./push.py                              # walk the queue: y / n / e(dit) / q
./push.py --yes                        # file everything pending, no prompts
./push.py --only f84e7312              # just that one

e opens the proposal in $EDITOR; save and it re-displays with your changes before you decide. Filed proposals move to filed/ with the server's response recorded in them, so the queue only ever shows what is still outstanding.

The api_token is the same one /api/health-sync uses — the value in the Pi's config.json, which is also what you type into SardinesSync's settings.

When a document is already filed and the proposal describes it differently, push shows the difference field by field and offers to correct the filed copy:

  already filed as #12, and this differs:
    date
      filed    2026-01-11
      proposed 2026-01-14
  update the filed copy? [y/N]

Only fields the proposal actually has are offered. A proposal with no provider means "could not determine one", not "there is no provider" — it will never blank a value you typed into the web interface. --yes deliberately does not overwrite; --update-duplicates is the flag that says do it without asking.

Two safety properties worth knowing. push.py re-hashes the PDF before sending and refuses if it no longer matches the proposal, because a replaced file would otherwise be filed under a description written for different bytes. And the server dedupes on that hash, returning duplicate: true rather than filing a second copy — which is what makes --yes safe to re-run after a dropped connection.

./triage.py                 # triage the inbox, ask qwen for descriptions
./triage.py --dry-run       # gates 1-2 only, no model, fast — use this while tuning
./triage.py --dry-run -v    # ...and show every signal that fired
./triage.py --force         # re-triage documents already in the manifest
./triage.py --report        # list what is sitting in the queue
./triage.py --limit 3       # stop after three new documents
./triage.py --inbox ~/else  # scan somewhere other than ~/Downloads/taildrop

--dry-run needs no Ollama and takes milliseconds. It is the loop to work in while adjusting signal weights; only spend the 90 seconds once the classification is right.

If the server sits behind Cloudflare, it answers urllib's default Python-urllib/3.x User-Agent with 403 Forbidden — before the request reaches the application, so it reads like an auth failure and is not one. Both clients send a real User-Agent for this reason. A 403 where you expected 401 is the proxy, not your token.

• pdftotext (poppler)
• ollama serve with qwen-local — only for the describe step
• stdlib Python only, no venv

• state/manifest.json — sha256 → first seen, source name, verdict. Delete a line to make one document new again; delete the file to re-triage everything.
• queue/<sha[:16]>.json — one proposal per document. Non-clinical documents are recorded in the manifest but write no proposal.

• No OCR. A scanned PDF has no text layer, so it cannot be classified or described. Those come back needs_ocr and are skipped rather than silently dropped.
• Tuned on one person's inbox. 69 documents, one patient, a handful of Oklahoma health systems. The anchors should generalise; the veto thresholds are calibrated to this corpus and would need re-measuring on another.
• A duplicate POST ignores new metadata. Re-triaging a document that is already filed produces a better description that push.py will not apply — the server sees a known hash and returns duplicate: true. Correct default (an idempotent create should not silently rewrite the record), but there is no update path yet. Editing the document in the web UI is the workaround.

MIT — see LICENSE.