Multimodal agentic clinical summarization powered by MedGemma. Turns fragmented patient data — labs, vitals, imaging, medications, notes — into a structured, evidence-linked physician summary.

Live demo: https://chart-copilot-production.up.railway.app/ Login: doctor@example.com / password123

Four MedGemma specialist agents (lab, imaging, medication, vitals) analyze clinical inputs in parallel, then a Gemini coordinator synthesizes their outputs into a single structured summary:

• One-liner and ranked active problem list
• Red flags with severity, confidence, and source evidence
• Lab/vital trends and medication safety findings
• Guideline-compliance gaps (HFrEF, AF, T2DM)
• Missing critical data to collect next

Every finding links back to its source input for rapid clinician verification.

Two-service monorepo (packages/frontend + packages/backend). MedGemma runs as an external HF endpoint, not a local service.

• Node.js >= 20
• A Google AI Studio API key (for Gemini)
• (Optional) A HuggingFace token + MedGemma Inference Endpoint for full MedGemma mode

npm install

Create a .env file in packages/backend/:

# Required
GEMINI_API_KEY=your_gemini_api_key

# Optional — enables MedGemma specialist agents
# Without these, specialists fall back to Gemini
HF_TOKEN=your_huggingface_token
MEDGEMMA_URL=https://your-endpoint.endpoints.huggingface.cloud

# Optional — auto-generated if not set
JWT_SECRET=any_random_string

npm run seed

This creates one comprehensive demo case (CHF + AKI + hyperkalemia + AF + T2DM) with 15 clinical inputs across 5 modalities.

Both frontend and backend in parallel:

npm run dev

Or individually:

npm run dev:backend    # Express API on http://localhost:3001
npm run dev:frontend   # Vite dev server on http://localhost:5173

Open the frontend, log in with doctor@example.com / password123, select the demo patient, and hit Compile Summary. The system runs all 4 specialists + coordinator and produces a full structured summary (typically 90–150 seconds).

If you don't have a MedGemma endpoint, the system works entirely on Gemini. Set in your .env:

USE_MEDGEMMA_AGENTS=false

Or just omit HF_TOKEN — the circuit breaker will detect the unavailable endpoint and fall back to Gemini automatically.

npm run build                          # Build all packages
npm run test -w packages/backend       # Run backend tests

packages/
  backend/
    src/
      agents/          # Specialist + coordinator agent definitions
      config/          # LLM, compilation, and app config
      db/              # SQLite schema, migrations, seed data
      fhir/            # FHIR R4 bundle export
      middleware/      # Auth (JWT), error handling
      routes/          # REST API routes
      services/        # Compile pipeline, MedGemma client, persistence
      tools/           # 13 clinical decision tools (scoring, cross-ref, history)
  frontend/
    src/
      features/        # Patient summary, clinical inputs, auth
      components/      # Shared UI components
      services/        # API client, FHIR export

• Pre-fetch, not ReAct: Specialist context is batch-fetched in a single Promise.allSettled (up to 11 tools) and injected into prompts, avoiding multi-turn overhead on the GPU endpoint.
• Gemini function calling for coordinator: The coordinator uses native multi-turn function calling (up to 10 rounds) for cross-domain reasoning — guideline checks, risk scoring, context queries.
• 5-strategy JSON parser + Gemini repair: Handles the full spectrum of malformed LLM output (fenced blocks, trailing commas, truncated JSON, thinking tokens).
• Circuit breaker + semaphore: Protects the GPU endpoint with failure detection (2-failure threshold, 120s cooldown) and concurrency control (max 2 parallel requests).
• Atomic persistence: All summary artifacts write to 13 normalized tables in a single SQLite transaction — all or nothing.

Chart Copilot is a decision-support draft assistant for demonstration and research use only. It does not replace clinician judgment or institutional care workflows.