flowchart LR
    subgraph SRC["Data Sources"]
        O["Orders\n3 platforms\n500/min"]
        S["IoT Sensors\n2,000/min"]
        G["GPS Pings\n5,000/min"]
        M["Menu CDC\n200/day"]
        F["Feedback\nBatch CSV"]
    end

    subgraph SPEED["⚡ Streaming Path — Speed Layer"]
        K[Apache Kafka]
        SS[Spark Structured\nStreaming]
        B1[Bronze\nDelta Lake]
        S1[Silver\nData Vault]
        G1["Gold\nStar Schema\n(approximate)"]
    end

    subgraph BATCH["📦 Batch Path — Truth Layer"]
        AF[Airflow\nDaily 4AM]
        SB[Spark Batch]
        S2[Silver\nData Vault]
        G2["Gold\nStar Schema\n(exact)"]
    end

    subgraph SERVE["Serving"]
        MB[Metabase]
        DDB[DynamoDB API]
        RS[Redshift SQL]
    end

    O --> K
    S --> K
    G --> K
    M --> K
    K --> SS --> B1 --> S1 --> G1

    F --> AF
    B1 --> AF --> SB --> S2 --> G2

    G2 -- "Reconciliation 6AM\nbatch overwrites streaming" --> G1

    G1 --> MB
    G1 --> DDB
    G1 --> RS

Why Lambda here? Orders transition through states: placed → confirmed → preparing → ready → picked_up → delivered (or cancelled at any stage). A delivered order might be refunded 2 hours later, changing final revenue. The streaming path gives dashboards fast-but-approximate numbers. The daily batch path reprocesses 48 hours of Bronze to produce exact numbers and overwrites the streaming Gold. Delta Lake MERGE makes this atomic overwrite safe.

Three platforms define the same entities differently:

Star Schema would force premature alignment. Data Vault solves this:

• Hubs unify identity — one hub_customer row per real person, regardless of platform
• Satellites preserve source-specific attributes with full SCD2 history
• Links capture relationships (which order → which customer → which kitchen+brand)

Gold then transforms Data Vault → Star Schema for analyst-friendly 2-table joins.

Key tables:

HUBS                    LINKS                       SATELLITES
hub_customer            link_order_customer          sat_customer_profile (SCD2)
hub_order               link_order_kitchen_brand     sat_order_details
hub_kitchen                                          sat_order_status (state machine)
hub_menu_item                                        sat_menu_item_details (SCD2)

customer_identity_bridge
├── platform: uber_eats / doordash / own_app
├── platform_customer_id: ue_cust_44821
├── email_hash: MD5(normalized email)
├── phone_hash: MD5(phone) — nullable
├── match_confidence: 1.0 (exact) ... 0.6 (fuzzy)
└── match_method: exact_email / fuzzy_name_address / manual_override

Algorithm:

1. Normalize emails (lowercase, trim)
2. Group all platform IDs by email_hash
3. Assign one customer_key per email group
4. Null emails → fuzzy match on name + delivery_address
5. Store match_method + confidence for auditability

A customer ordering from Uber Eats AND DoorDash AND the direct app resolves to one row in dim_customer. Their cross-platform behavior is now visible: average order value per platform, platform loyalty, cannibalization analysis.

Dual-grain strategy: Atomic events in Silver for correctness + pre-aggregated in Gold for performance. Every fact table has a deliberate grain decision documented.

SCD types used: Type 0 (zones — never change) · Type 1 (kitchens — overwrite) · Type 2 (menu items, customers — full history)

Bridge tables: bridge_kitchen_brand (M:M — one kitchen runs multiple brands) · customer_identity_bridge (cross-platform resolution)

flowchart LR
    subgraph SRC["Data Sources"]
        W["Wearable Devices\n4 types"]
        P["Pharmacy\nCDC Events"]
        E["EHR\nFHIR Batches"]
    end

    subgraph KAPPA["🔄 Single Streaming Pipeline"]
        K[Apache Kafka]
        SS["Spark Structured\nStreaming"]
        B[Bronze\nDelta Lake]
        SL["Silver\nNormalized\nMetric Explosion"]
        GL["Gold\nSnowflake Schema"]
    end

    subgraph MAINT["🔧 Airflow — Maintenance Only"]
        IR["Identity Resolution\nDaily 2AM"]
        DQ["Data Quality\nEvery 6h"]
        CM["Compaction\nDaily"]
        HA["HIPAA Audit\nWeekly"]
    end

    subgraph SERVE["Serving"]
        SU[Superset]
        CO[Cosmos DB API]
        AZ[Azure Functions\nAlerts]
    end

    W --> K
    P --> K
    K --> SS --> B --> SL --> GL
    E --> B

    SL -.-> IR
    SL -.-> DQ
    GL -.-> CM
    GL -.-> HA

    GL --> SU
    GL --> CO
    GL --> AZ

Why Kappa here? A heart rate reading of 72 bpm at 2:34pm doesn't get "corrected" later. It's an immutable measurement. No need for Lambda's dual-pipeline complexity. One streaming engine handles real-time, late-arriving data (30% of events arrive hours late from batch sync), AND backfills (reset Kafka offsets, replay through same code). Same pipeline, same logic, one truth.

Airflow's role is completely different here. In GhostKitchen, Airflow is the conductor — it orchestrates the batch ETL path. In PulseTrack, Airflow is the janitor — it handles maintenance (identity resolution refresh, data quality sweeps, Delta compaction, HIPAA audits) while the streaming pipeline runs continuously on its own.

Bronze stores one row per device reading with nested metrics:

{"device_id": "SW-001", "metrics": {"heart_rate_bpm": 72, "spo2_pct": 98, "hrv_ms": 45}}

Silver explodes this to one row per metric:

SW-001 | heart_rate_bpm | 72.0 | ✅ valid
SW-001 | spo2_pct       | 98.0 | ✅ valid
SW-001 | hrv_ms         | 45.0 | ✅ valid

Why? Anomaly detection, dashboards, and aggregations all operate on individual metrics. Nested JSON forces complex extraction on every query. Explosion is done once in Silver, benefitting every downstream consumer.

Quality flags (not dropped):

Invalid readings are never dropped — they're flagged and kept because out-of-range values may indicate genuine medical events that anomaly detection should evaluate.

Healthcare dimensions are deeply hierarchical:

• ICD-10 codes: Chapter → Category → Specific Code (3 levels)
• Medications: Drug Class → Medication → Dosage → Patient Assignment
• Devices: Device Type → Firmware Version → Patient Reassignment

Star Schema would create a "kitchen sink" dim_patient with 50+ columns. Snowflake normalizes these hierarchies into separate dimension tables with foreign keys.

Fact tables:

Dimension tables with SCD types:

The hardest engineering problem in PulseTrack. A single patient may appear as:

device_account_id  →  email  →  hospital_mrn  →  pharmacy_id  →  insurance_id  →  phone_hash  →  ssn_hash

Multi-hop graph matching: device_account links to email, email links to hospital_mrn, hospital_mrn links to pharmacy_id — all resolving to one unified patient_key via the patient_identity_bridge table (1,159 rows for 100K user dataset).

This is significantly more complex than GhostKitchen's 3-platform identity resolution because healthcare identifiers are fragmented across completely independent systems (hospitals, pharmacies, insurers, device manufacturers) with no shared login.

A runner with resting HR 52 showing HR 85 is more concerning than a sedentary person at HR 85. Population-level thresholds miss this.

Per-patient rolling statistics (mean, σ for each metric over 30 days). New reading compared against personal baseline. Deviation > 2σ = anomaly alert.

• State: Keyed by (patient_key, metric_name). Each key stores running stats + circular buffer.
• Backed by: RocksDB state store in Spark Structured Streaming.
• TTL: 90 days for inactive patients.
• Output: fact_anomaly_alert in Gold → Azure Functions → email/SMS notifications.

• De-identification: Gold uses age_bracket (not exact age), city (not address), no names
• Encryption: Delta Lake files encrypted at rest
• Access control: data_engineer (all layers), clinical_analyst (Gold only, PII masked), researcher (Gold, fully de-identified)
• Deletion: Delta Lake DELETE across all layers → verification sweep → audit log
• Retention: HIPAA requires 7-year retention. Patient deletion = de-identify PII but keep analytical record.
• Audit: Weekly dag_hipaa_audit scans for PII leakage across all layers

flowchart TD
    INPUT["Patient Vitals + Clinical Notes + FHIR Bundle"]
    INPUT --> S1["Stage 1: NER Extraction\n33 meds · 28 conditions · 25 procedures"]
    S1 --> S2["Stage 2: Knowledge Graph\nNetworkX DiGraph · 3 edge types\n20 treatment · 5 contraindication · 9 diagnostic"]
    S1 --> S3
    S2 --> S3["Stage 3: Multi-Agent Reasoning"]

    subgraph AGENTS["Three Agents — Sequential"]
        T["Triage Agent\nESI v4 · 6 vital thresholds"]
        D["Diagnostic Agent\nICD-10 · 6 complaint categories"]
        TR["Treatment Agent\n8 protocols · drug interactions\nallergy cross-reference"]
        T --> D --> TR
    end

    S3 --> AGENTS
    AGENTS --> S4["Stage 4: Risk Prediction\nRandom Forest + SHAP\n9 features · explainable"]
    AGENTS --> S5["Stage 5: RAG Retrieval\nFAISS + MiniLM-L6-v2\n15 curated guidelines"]
    S5 --> S6["Stage 6: Clinical Summarization\nT5 Transformer · beam search\nauto-fallback to pretrained"]
    S6 --> S7["Stage 7: Safety Evaluation\n4-axis LLM-as-Judge\nfactual · hallucination · accuracy · safety"]
    S7 -- "score < threshold" --> S6
    S7 --> S8["Stage 8: FHIR R4 Export\nPatient · Observation · RiskAssessment\nLOINC-coded vitals"]
    S4 --> S9
    S8 --> S9["Stage 9: Data Engineering\n7 integrated systems"]

One POST /api/command-center runs all 9 stages per patient encounter. No manual orchestration. Stage 7 (Safety Evaluation) has a feedback loop — if the score is below threshold, it triggers re-summarization with RAG context injection.

Consensus score: 0.3 × risk + 0.4 × confidence + 0.3 × evidence_grade

Clinical reasoning is multi-step. A single LLM call can't provide the auditability, separation of concerns, and protocol adherence required in healthcare.

The orchestrator chains all three and records a full audit trail (JSONB in PostgreSQL). Every decision is traceable.

Key design decision: RAG over fine-tuning for medical knowledge. Guidelines change — RAG enables hot-swap without retraining, and citations provide provenance.

Each system is ~200–500 lines of focused Python implementing production-faithful patterns:

Why custom over Airflow/Kafka/dbt? Zero external dependencies. Production-faithful patterns that run with just pip install. Demonstrates understanding of the internals, not just tool configuration.

ETL DAG execution flow:

ingest → validate → extract_entities → build_kg
                  → run_triage → run_diagnosis → run_treatment
                  → predict_risk
                                  run_diagnosis → retrieve_rag → generate_summary → evaluate_safety
                                                                                          ↓
export_fhir ← [run_treatment, predict_risk, generate_summary, evaluate_safety]
    ├── load_warehouse → emit_cdc → update_catalog
    └── track_lineage → update_catalog

ROUGE can't detect hallucinations or clinically dangerous errors. HERA uses a 4-axis evaluation:

Overall score = weighted sum. If below threshold → feedback loop re-triggers summarization with RAG context.

Deployment pipeline:

Push to main → GitHub Actions CI (lint + 104 tests + Docker build) → ECR → AWS App Runner auto-deploys

Observability (Prometheus + Grafana):

• 17-panel Grafana dashboard (auto-provisioned): System Health, Clinical Risk & Triage, API Performance, Pipeline & Data Engineering
• Prometheus scrapes /metrics every 5s: hera_requests_total, hera_request_failures_total, hera_request_latency_seconds
• Alert rule: TooManyHighRisk fires when high_risk_predictions > 3 for 30s

PostgreSQL (5 tables): patient_predictions, summaries, clinical_reasoning_sessions, ner_extractions, evaluation_reports — all populated automatically per API call.

Middleware: API key auth + rate limiting (30/min anon, 60/min auth) + HIPAA audit trail with trace IDs.

Cost: AWS App Runner 1 vCPU, 2GB RAM, auto-scales to 0 → ~$5–7/mo.