Medical Form Understanding & Population Pipeline

Healthcare administrative burden is a significant problem. Clinicians spend hours transferring patient information from clinical documents (SOAP notes, lab results, referrals) into standardized forms. This manual process is error-prone, time-consuming, and pulls skilled professionals away from patient care.

The challenge isn't just OCR or basic extraction, it's understanding context. A patient name in a SOAP note might be abbreviated ("Peter F."), while the same name in structured JSON is complete ("Peter Julius Fern"). A form might expect "Last, First, Middle Initial" format. The system needs to reason about which source to trust, how to normalize values, and when to admit uncertainty.

This project demonstrates that a locally-runnable, AI-assisted pipeline can meaningfully automate this workflow while maintaining the transparency and auditability that healthcare requires.

What I chose to solve:

• End-to-end pipeline from raw documents to populated PDF form
• Schema extraction from both fillable PDFs (AcroForm parsing) and scanned PDFs (OCR + layout heuristics)
• Multi-source information extraction with confidence scoring
• AI-assisted reconciliation when sources conflict or formats differ
• Evaluation framework to measure accuracy and detect hallucinations

What I scoped out:

• Complex table extraction (medications handled via regex, not full table parsing)
• Multi-page form support (focused on the single-page form provided)
• Handwriting recognition (lab_result.pdf was native text, not scanned handwriting)
• Production-grade error handling and logging

Why these choices: Given the 4-6 hour target, I prioritized demonstrating the full pipeline over perfecting individual components. A reliable working end-to-end slice is more valuable than a polished extraction module that never populates a PDF.

┌─────────────────────────────────────────────────────────────────────┐
│  Phase 1: Schema Extraction                                         │
│  - AcroForm widget parsing (fillable PDF)                          │
│  - OCR + layout heuristics (scanned PDF)                           │
│  - Output: schema.json (22 fields)                                 │
└─────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────┐
│  Phase 2: Document Ingestion                                        │
│  - JSON: Direct structured parsing                                  │
│  - TXT: SOAP section extraction (Subjective/Objective/etc.)        │
│  - PDF: PyMuPDF text extraction with OCR fallback                  │
└─────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────┐
│  Phase 3: Information Extraction                                    │
│  - JSON lookup (highest confidence)                                 │
│  - Regex patterns (dates, phones, medications)                     │
│  - LLM extraction (diagnoses, names from unstructured text)        │
└─────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────┐
│  Phase 4: Reconciliation & Confidence Scoring                       │
│  - Multi-factor confidence: method prior, format validity,         │
│    quote verification, source reliability                          │
│  - LLM value selection (prefer complete values)                    │
│  - LLM format transformation (name → "Last, First M.")             │
│  - Output: answers.json                                            │
└─────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────┐
│  Phase 5: PDF Population                                            │
│  - Schema-driven field mapping                                      │
│  - Multi-part field handling (phone: area/prefix/line)             │
│  - Confidence gating (threshold: 0.5)                              │
│  - Output: populated.pdf                                           │
└─────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────┐
│  Phase 6: Evaluation                                                │
│  - Accuracy vs. ground truth                                        │
│  - Hallucination detection (quote verification)                    │
│  - Confidence calibration analysis                                 │
│  - Format consistency checks                                       │
└─────────────────────────────────────────────────────────────────────┘

Key Design Decisions:

1. Method Confidence as Prior — Each extraction method (JSON lookup, regex, LLM) carries a reliability prior. JSON gets 1.0, regex 0.90, LLM 0.80. This prior is combined with runtime signals (format validity, quote verification) to compute final confidence.

2. Quote Verification for Hallucination Control — Every LLM extraction must provide a verbatim quote. We verify this quote exists in the source document. If not found, confidence is penalized.

3. Multi-Candidate Extraction — For key fields, we extract from multiple sources (e.g., DOB from both demographics.json and lab_result.pdf). When values agree, we apply a corroboration bonus (+0.10).

4. LLM Format Transformation — Rather than hardcoding name parsing rules, we use the LLM to transform "Peter Julius Fern" into "Fern, Peter J." based on the form field's label ("Last, First, Middle Initial").

Judicious AI Use: The pipeline uses LLM only where symbolic methods fall short. Phone number normalization uses regex (deterministic, fast). Diagnosis extraction from clinical notes uses LLM (requires reasoning). This hybrid approach balances reliability with capability.

Model: Mistral 7B via Ollama (local inference)

Why Mistral: Good balance of capability and speed for structured extraction tasks. Runs locally without API dependencies. Temperature set to 0 for deterministic outputs.

Prompt Design Principles:

1. Structured JSON Output — All prompts request JSON responses with specific schema. This enables programmatic parsing and validation.

2. Anti-Hallucination Constraints — Prompts explicitly require verbatim quotes from source text. Example:

   RULES:
   1. Only extract information EXPLICITLY stated in the context
   2. You MUST provide a verbatim quote as evidence
   3. If not found, respond with "NOT_FOUND"
   

3. Confidence Capping — LLM self-reported confidence is unreliable, so we cap it at 0.85 regardless of what the model claims.

4. Context Limiting — Source text is truncated to ~2000 characters to stay within context limits and reduce noise.

Key Observations:

• DOB achieved 1.00 confidence due to corroboration bonus (appeared in 2 sources)
• Name field required LLM format transformation to match form requirements
• 13 fields had no data in source documents (employment info, insurance details) — correctly marked with confidence 0

1. Ollama Response Format Changes — The Ollama Python library (v0.2+) returns Pydantic models instead of dicts. Initial code assumed dict access and failed silently. Solution: Added get_llm_response_text() helper to handle both formats.

2. Nested JSON Parsing from LLM — Simple regex \{[^}]+\} couldn't match nested JSON objects in LLM responses. Solution: Changed to \{[\s\S]*\} pattern and added multi-method parsing fallback.

3. Malformed Source Data — demographics.json contained a malformed phone number (613-6565-890 — 10 digits but wrong grouping). Solution: Parse all digits, apply NANP format, flag with reduced confidence (0.95 instead of 1.0).

4. Multi-Value Fields Misinterpreted as Conflicts — Secondary diagnoses (Hypertension, GERD, Hyperlipidemia) were initially treated as conflicting single-value candidates. Solution: Aggregate multi-value fields into comma-separated strings.

5. PDF Form Field Mapping — Form fields like "Diagnosis_Secondary1" and "Diagnosis_Secondary2" suggested splitting values, but this lost data when we had 3+ diagnoses. Solution: Put all values in first field, enable PDF handled text wrapping.

6. Name Format Mismatch — Form expected "Last, First, Middle Initial" but sources provided "First Middle Last". Solution: LLM-based format transformation rather than hardcoded parsing rules.

1. Vision Model for Scanned Forms — Replace Tesseract OCR with a vision-language model (e.g., LLaVA) for better handling of complex layouts, handwritten annotations, and checkbox detection.

2. Confidence Calibration Tuning — Current weights (method prior 30%, format validity 25%, quote verification 25%, source reliability 20%) were set heuristically. Would benefit from empirical tuning on a larger dataset.

3. Ensemble LLM Extraction — For high-stakes fields (diagnoses, medications), run extraction with multiple prompts or models and use voting/agreement to boost confidence.

4. Active Learning Loop — Collect human corrections on low-confidence fields and use them to fine-tune extraction prompts or train field-specific classifiers.

5. Multi-Form Generalization — Test schema extraction on different form types (insurance claims, referral letters) to validate the pipeline generalizes beyond the specific form provided.

6. Medication Interaction Warnings — Integrate a drug database to flag potential interactions or contraindications when populating medication fields.

# Prerequisites
pip install pymupdf pytesseract pdf2image pypdf ollama python-dateutil
# See full guidance on certain installations and package placements in notebook itself
# See full guidance on Tesseract, Ollama, Poppler executable location variables setting in notebook itself

# System dependencies (Windows)
# - Tesseract OCR: https://github.com/tesseract-ocr/tesseract
# - Poppler: https://github.com/oschwartz10612/poppler-windows/releases
# - Ollama: https://ollama.ai (run `ollama pull mistral`)

# Execute
jupyter notebook main.ipynb
# Run all cells sequentially

Attached requirements.txt is an optional way to setup the environment, however Poppler, Tesseract and Ollama installations are necessary to handle outside of that still, depending on OS used at the time.

Outputs:

• outputs/schema.json — Form field schema
• outputs/answers.json — Extracted values with confidence and reasoning
• outputs/populated.pdf — Filled form

Author: Kutlu Mizrak | January 2026