Ilios API

graph TB
    subgraph CORE["Ilios API Server (Bun Runtime)"]
        API[Hono API<br/>Routes & Middleware<br/>Bun Native I/O]
        JP[Job Processor<br/>Worker Manager<br/>IPC Communication]
        DB[(SQLite DB<br/>./data/ilios.db<br/>WAL Mode<br/>SQLITE_BUSY Retry)]
        
        subgraph "Worker Threads (Bun Worker)"
            W0{{Worker 0<br/>postMessage IPC<br/>Atomic Claim<br/>Shared DB<br/>Retry Logic}}
            W1{{Worker 1<br/>postMessage IPC<br/>Atomic Claim<br/>Shared DB<br/>Retry Logic}}
        end
        
        API -.-> JP
        API -->|Read/Write<br/>WAL Mode| DB
        JP <-->|postMessage<br/>2-241x faster| W0
        JP <-->|postMessage<br/>2-241x faster| W1
        JP -->|Cleanup Jobs| DB
        
        W0 -->|Atomic Claim<br/>withRetry helper| DB
        W1 -->|Atomic Claim<br/>withRetry helper| DB
    end
    
    subgraph EXT["External Cloud Services"]
        EXT_S3[("☁️ S3 Storage<br/>(Tigris)<br/>Bun.write streaming")]
        EXT_MISTRAL[("🤖 Mistral OCR<br/>API")]
    end
    
    API -->|Upload Files<br/>Multipart >50MB| EXT_S3
    W0 -->|Bun.write Download<br/>Zero-Copy| EXT_S3
    W1 -->|Bun.write Download<br/>Zero-Copy| EXT_S3
    
    W0 -->|OCR Request| EXT_MISTRAL
    W1 -->|OCR Request| EXT_MISTRAL
    
    style CORE fill:#e3f2fd,stroke:#1976d2,stroke-width:3px,color:#000
    style EXT fill:#fce4ec,stroke:#c2185b,stroke-width:3px,color:#000
    style DB fill:#e1f5ff,stroke:#0288d1,stroke-width:3px,color:#000
    style API fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
    style JP fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000
    style W0 fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#000
    style W1 fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#000
    style EXT_S3 fill:#ffebee,stroke:#c2185b,stroke-width:2px,color:#000
    style EXT_MISTRAL fill:#ffebee,stroke:#c2185b,stroke-width:2px,color:#000

sequenceDiagram
    participant C as Client
    participant API as Hono API<br/>(Bun)
    participant DB as SQLite DB<br/>(WAL Mode)
    participant JP as Job Processor<br/>(Worker Manager)
    participant W as Worker Thread<br/>(Bun Worker)
    participant S3 as S3 Storage<br/>(Tigris)
    participant M as Mistral OCR

    Note over C,API: Document Submission
    C->>API: POST /api/documents/submit<br/>(file, retentionDays)
    API->>API: Detect MIME type
    API->>S3: Upload file (Bun.write)<br/>multipart if >50MB
    API->>DB: INSERT document (status=pending)
    API->>DB: INSERT job (type=convert, status=pending)
    API-->>C: 202 Accepted {id, status: pending}

    Note over JP,W: Async Job Processing (Worker Threads)
    JP->>DB: Check for pending jobs<br/>(count pending)
    JP->>W: postMessage({type: 'process'})
    
    W->>DB: BEGIN TRANSACTION<br/>(with retry on SQLITE_BUSY)
    W->>DB: SELECT pending job<br/>(ORDER BY priority, LIMIT 1)
    W->>DB: UPDATE job SET status=processing,<br/>worker_id=W, attempts++
    W->>DB: COMMIT (atomic claim)
    
    alt Job Claimed Successfully
        W->>DB: UPDATE document SET status=processing
        W->>S3: Bun.write(tempPath, s3File)<br/>Zero-copy streaming for >100MB
        W->>W: Bun.file().arrayBuffer()<br/>Direct processing <100MB
        W->>M: POST /v1/files/upload + OCR<br/>(Uint8Array buffer)
        M-->>W: {pages[], markdown, usage}
        W->>DB: UPDATE document SET<br/>content=markdown, status=completed
        W->>DB: INSERT usage record<br/>(tokens, cost)
        W->>DB: UPDATE job SET<br/>status=completed, completedAt=now
        W->>W: Clean up temp files
        W-->>JP: postMessage({type: 'completed', jobId})
    else Job Processing Failed
        W->>DB: failJob(id, error)<br/>(with retry logic)
        alt attempts < maxAttempts
            W->>DB: UPDATE job SET status=pending,<br/>scheduledAt=now+backoff
            Note over W,DB: Exponential backoff<br/>(100ms, 200ms, 400ms, 800ms)
        else attempts >= maxAttempts
            W->>DB: UPDATE job SET status=failed,<br/>completedAt=now
            W->>DB: UPDATE document SET status=failed
        end
        W-->>JP: postMessage({type: 'failed', jobId, error})
    end

    Note over C,API: Status Check & Download
    C->>API: GET /api/documents/status/{id}
    API->>DB: SELECT document WHERE id={id}
    API-->>C: {status, error?, metadata?}

    C->>API: GET /api/documents/{id}
    API->>DB: SELECT content WHERE id={id}
    alt status=completed
        API-->>C: 200 OK (markdown text)
    else status=processing
        API-->>C: 202 Accepted {status: processing}
    else status=failed
        API-->>C: 500 Error {error}
    end

stateDiagram-v2
    [*] --> pending: Job Created<br/>(scheduledAt=now)
    pending --> processing: Worker Claims (atomic TX)<br/>with withRetry() helper

    processing --> completed: Success<br/>Sends completion message
    processing --> pending: Job Timeout<br/>(>5min, attempts < max)<br/>Exponential backoff
    processing --> failed: Job Timeout<br/>(>5min, attempts >= max)
    processing --> pending: Error + Retry<br/>(attempts < max)<br/>scheduledAt=now+backoff
    processing --> failed: Error + No Retry<br/>(attempts >= max)<br/>Sends failure message

    completed --> [*]
    failed --> [*]

    note right of processing
        Worker Thread (Bun Worker API)
        - True OS-level threads
        - postMessage 2-241x faster IPC
        - Own DB connection per thread
        - withRetry on all writes
        - No heartbeat mechanism
        - Cleanup runs every 60s
        - Job timeout-based orphan detection
    end note

erDiagram
    documents {
        text id PK
        text file_name
        text mime_type
        integer file_size
        text s3_key
        text content
        json metadata
        text status "pending|processing|completed|failed|archived"
        text error
        timestamp created_at
        timestamp processed_at
        timestamp archived_at
        integer retention_days
        text user_id
        text api_key
    }
    
    usage {
        text id PK
        text document_id FK
        text user_id
        text api_key
        text operation
        integer input_tokens
        integer output_tokens
        integer base_cost_cents
        integer margin_rate
        integer total_cost_cents
        timestamp created_at
    }
    
    jobQueue {
        text id PK
        text document_id FK
        text type
        text status "pending|processing|completed|failed|retrying"
        integer priority
        integer attempts
        integer max_attempts
        json payload
        json result
        text error
        text worker_id FK
        timestamp scheduled_at
        timestamp started_at
        timestamp completed_at
        timestamp created_at
    }
    
    batches {
        text id PK
        text user_id
        text api_key
        integer total_documents
        integer completed_documents
        integer failed_documents
        text status "pending|processing|completed|failed"
        integer priority
        timestamp created_at
        timestamp completed_at
        json metadata
    }
    
    documents ||--o{ usage : "has"
    documents ||--o{ jobQueue : "has"
    documents }o--|| batches : "belongs to"