Single-host, isolation-first GenAI stack for sensitive data processing. This repository is for running local chat, embedding, and STT backends with strict network isolation and pinned egress exceptions only where explicitly configured.
Read the disclaimer and security risk assessment first.
By cloning, forking, or running this software, you acknowledge that you have read this disclaimer and agree to use the software at your own risk.
If you use this repository or its architecture in your research, please cite our paper:
Secure On-Premise Deployment of Open-Weights Large Language Models in Radiology: An Isolation-First Architecture with Prospective Pilot Evaluation
Sebastian Nowak*, Jann-Frederick Laß*, Narine Mesropyan, Babak Salam, Nico Piel, Wolfgang Block, Alois Martin Sprinkart, Julian Alexander Luetkens, Benjamin Wulff*, Alexander Isaak*
* contributed equally
arXiv preprint, 2026. (arXiv ID coming soon)
Set up the local Python environment once. This creates ./.venv/bin/python for the repository-specific interpreter:
python3 -m venv .venv
./.venv/bin/python -m pip install -U pip
./.venv/bin/python -m pip install -r requirements.txtFastest safe path:
python3 start.py wizard
python3 start.py upWhat the wizard does:
- writes non-secret settings to
.env - keeps secrets out of
.env - helps you choose runtime mode, optional features/apps, and backend deployments
Before startup, make sure model artifacts already exist locally. Normal runtime downloads are not part of the intended path.
The tree below is a guided overview of the main tracked files and folders. Runtime-generated artifacts such as compose/generated/ are intentionally omitted.
.
├── README.md # Main project overview and quick start
├── LICENSE # License terms for the repository
├── .gitignore # Ignore rules for local and generated artifacts
├── requirements.txt # Shared Python dependencies for CLI tools and tests
├── pytest.ini # Pytest collection, path, and marker configuration
├── start.py # Main CLI for the wizard, validation, and startup
├── stop.py # Shutdown helper for the Compose project
├── apps/ # App-specific folders for optional add-ons
│ ├── dictation/ # Gradio speech-to-text app exposed via ingress
│ │ ├── app.py # Dictation application entrypoint
│ │ ├── Dockerfile # Image build recipe for the dictation app
│ │ ├── requirements.txt # Python dependencies for the dictation app
│ │ ├── translate_prompt.txt # Prompt template used during transcription flow
│ │ └── README.md # Dictation-specific usage notes
│ ├── cohort_feasibility/ # Placeholder folder for the cohort feasibility app
│ │ └── README.md # Temporary note until app materials are published
│ ├── dataset_structuring/ # Placeholder folder for the dataset structuring app
│ │ └── README.md # Temporary note until app materials are published
│ └── icd_10_coding/ # Placeholder folder for the ICD-10 coding app
│ └── README.md # Temporary note until app materials are published
├── compose/ # Compose base files and overlays that assemble the stack
│ ├── base.yml # Core services, networks, and baseline runtime wiring
│ ├── hardening.yml # Shared hardening defaults extended by other services
│ ├── model.base.yml # Common worker template for rendered model deployments
│ ├── schema.toml # Central env schema used by the wizard and docs tooling
│ ├── modes/ # Runtime-mode overlays
│ │ ├── frontend.provider.yml # Chatbot-provider mode with frontend-facing services
│ │ └── batch.client.yml # Batch-client mode for localhost-only API access
│ ├── features/ # Optional feature overlays layered onto a mode
│ │ ├── ldap.yml # LDAPS egress proxy overlay used by frontend directory authentication
│ │ ├── api_egress.yml # Pinned HTTPS egress proxy overlay for upstream APIs in batch mode
│ │ ├── metrics.yml # Prometheus, Grafana, and exporter services overlay
│ │ ├── chat_purger.yml # Automated OpenWebUI chat retention cleanup
│ │ ├── rate_limiting.yml # OpenWebUI Pipelines overlay for adaptive rate limiting
│ │ ├── embedding_backend.yml # Marker overlay selected when an optional embedding backend is enabled
│ │ ├── stt_backend.yml # Marker overlay selected when an optional STT backend is enabled
│ │ └── dmz_egress.yml # Defines the routed `dmz_egress` network used by explicit airlocks
│ ├── apps/ # Compose overlays for optional application services
│ │ ├── dictation.yml # Dictation app service overlay
│ │ ├── cohort_feasibility.yml # Cohort feasibility app service overlay
│ │ └── dataset_structuring.yml # Dataset structuring utility app overlay
│ └── models/ # Model catalog and per-family compose templates
│ ├── README.md # Model zoo reference and deployment conventions
│ ├── backend_router.yml # Shared router used in front of model workers
│ ├── llm/ # LLM family templates and metadata
│ │ └── <family>/ # One folder per LLM family with `base.yml` and `model.toml`
│ ├── embedding/ # Embedding family templates and metadata
│ │ └── <family>/ # One folder per embedding family with `base.yml` and `model.toml`
│ ├── stt/ # STT family templates and metadata
│ │ └── <family>/ # One folder per STT family with `base.yml` and `model.toml`
│ └── testing/ # Mock/lightweight model families used by tests
│ └── <family>/ # Test-only fake backend definitions
├── docs/ # Operator, feature, app, and developer documentation
│ ├── README.md # Main operator documentation hub
│ ├── setup-basics.md # Common setup and prerequisite guide
│ ├── lifecycle.md # Runtime checks, logs, and shutdown guide
│ ├── disclaimer.md # Legal usage terms, warranty disclaimer, and liability limits
│ ├── risk_assessment.md # Security posture, trust boundaries, residual risks, and audit focus
│ ├── diagram.png # Architecture diagram shown in this README
│ ├── modes/ # Detailed runtime-mode documentation
│ │ ├── chatbot-provider.md # WebUI and HTTPS deployment guide
│ │ └── batch-client.md # Localhost API deployment guide
│ ├── features/ # Docs for optional stack features
│ │ └── *.md # One page per feature such as LDAP or metrics
│ ├── apps/ # Docs for optional apps
│ │ └── *.md # One page per app such as dictation
│ └── dev/ # Contributor docs for extending the stack
│ ├── README.md # Entry point for developer-facing docs
│ └── *.md # Guides for apps, model families, and extensions
├── images/ # Docker image build contexts and runtime helper assets
│ ├── nginx/ # Hardened Nginx images for ingress, routing, and egress
│ │ ├── ingress/ # Main inbound proxy config and templates
│ │ ├── backend_router/ # Internal router in front of model workers
│ │ ├── api_egress/ # Allowlisted egress proxy for upstream APIs
│ │ └── ldap_egress/ # Allowlisted egress proxy for LDAP traffic
│ ├── openwebui/ # Customized OpenWebUI images plus pipeline and retention helpers
│ │ ├── pipelines/ # Custom OpenWebUI Pipelines implementations
│ │ └── functions/ # Custom OpenWebUI Functions plugins
│ ├── metrics/ # Build assets for Prometheus, Grafana, and the Nginx metrics exporter
│ │ ├── prometheus/ # Prometheus image and generated-config template
│ │ ├── grafana/ # Grafana image and dashboard provisioning
│ │ └── exporter/ # Nginx Prometheus exporter image context
│ ├── vllm_worker/ # Main vLLM-based worker image for model-serving containers
│ └── dummy_vllm_worker/ # Dummy OpenAI-compatible worker image used by tests
├── security_helpers/ # Host and container helper scripts for isolation enforcement and monitoring
│ ├── apply_host_firewall.py # Applies host firewall rules for network isolation
│ ├── install_security_helpers.sh # Installs container-side helper scripts and minimal network tools into images
│ ├── active_isolation_monitoring_entrypoint.sh # Container entrypoint wrapper that monitors for isolation failures
│ ├── check_egress.sh # Container-side probe for validating blocked or allowed network reachability
│ ├── healthcheck.sh # Shared healthcheck script copied into hardened containers
│ └── nginx_debug_dump.sh # Container-side troubleshooting helper for Nginx runtime config
├── tests/ # Managed-stack integration and isolation test suite
│ ├── README.md # Testing model, markers, and common commands
│ ├── conftest.py # Shared fixtures that boot and tear down test stacks
│ ├── helpers/ # Test helpers for Docker, PKI, networking, and fixtures
│ ├── integration/ # End-to-end behavior tests across runtime modes
│ ├── isolation/ # Security-boundary and hardening regression tests
│ ├── model_deployments/ # Sample deployment TOMLs consumed by the test suite
│ ├── compose.test.*.yml # Test-only Compose fragments for helper containers and mocks
│ └── mock_api_server_entrypoint.sh # Entrypoint for the mock upstream API in tests
└── utils/ # Shared Python modules behind startup and generation tools
├── stack/ # Core stack orchestration logic used by the CLI
│ ├── startup.py # Wizard, validation, and Compose startup orchestration
│ ├── schema.py # Parser/loader for `compose/schema.toml`
│ └── deployments.py # Rendering and resolution of concrete model deployments
├── models/ # Model deployment parsing and topology helpers
│ └── deployment.py # TOML parser and resolver for deployment configs
└── scripts/ # Standalone helper scripts
├── build_docs.py # Generates docs from schema and model metadata
├── configure_env.py # Standalone entrypoint for the interactive `.env` wizard
└── create_localhost_pki.py # Creates localhost certificates for manual local HTTPS testing
- New operator: Operational docs hub
- Common prerequisites and backend selection: Common setup
- Tests: tests/README.md
- Legal and risk docs: Disclaimer and Security risk assessment
python3 start.py validate
python3 start.py wizard --start
python3 start.py up --env-file /path/to/.env
python3 stop.pySecrets can be exported in the shell or provided via *_FILE variables such as CERTIFICATE_KEY_FILE. Do not set both VAR and VAR_FILE for the same secret.