AI Survival

A fully offline, portable AI assistant on an external HDD/USB. Plug it into any machine, run one script, and chat with a local LLM through a browser-based UI. No internet required after initial setup.

• llama.cpp runs the LLM inference (OpenAI-compatible API)
• Open WebUI provides the browser chat interface
• Docker Compose or local Python serves both components
• GGUF models are stored on the HDD, loaded at startup

Two deployment modes — choose what suits your target machine:

External HDD
├── llama.cpp server  ←── serves the model (Docker or local binary)
├── Open WebUI        ←── chat UI at localhost:3000
└── models/*.gguf     ←── offline model files

# Linux / macOS
./scripts/setup.sh

# Windows
scripts\setup.bat

You will be prompted to choose Docker or Local mode. You can also skip the prompt:

./scripts/setup.sh --docker    # Docker mode
./scripts/setup.sh --local     # Local mode (no Docker)

./scripts/download-models.sh

# Linux / macOS
./scripts/start.sh              # Docker mode
./scripts/start-local.sh        # Local mode

# Windows
scripts\start.bat               # Docker mode
scripts\start-local.bat         # Local mode

Open http://localhost:3000 and start chatting.

./scripts/stop.sh               # Docker mode
./scripts/stop-local.sh         # Local mode
scripts\stop.bat                # Windows Docker
scripts\stop-local.bat          # Windows Local

Download interactively:

./scripts/download-models.sh          # Interactive menu
./scripts/download-models.sh --list   # List available models

Switch between downloaded models:

./scripts/switch-model.sh

For NVIDIA GPUs with CUDA:

./scripts/start.sh --gpu

Set GPU_LAYERS=-1 in .env to offload all layers to GPU. Requires NVIDIA Container Toolkit.

If the target machine doesn't have Docker, you can download the installer in advance:

./scripts/download-docker.sh             # Current platform
./scripts/download-docker.sh --all       # All platforms
./scripts/download-docker.sh --list      # List already downloaded

Installers are saved to installers/docker/:

The setup.sh script will detect missing Docker and point you to these files automatically — it does not block or exit if Docker is absent.

Local mode runs llama.cpp and Open WebUI as native processes on the host.

./scripts/install-local.sh           # Linux/macOS
scripts\install-local.bat             # Windows
./scripts/install-local.sh --check   # Check what is already installed

This:

1. Downloads and extracts the llama.cpp pre-built binary for your platform
2. Creates a Python venv at data/webui-venv/ and installs Open WebUI
3. Creates .env.local with local-mode settings

For fully offline local install, pre-download files before copying to the HDD:

install-local.sh checks these directories first before attempting to download.

See OFFLINE.md for the full guide — preparation, deployment, troubleshooting, and updating.

Summary:

1. Prepare (online): download models, build/save Docker images or cache local binaries
2. Copy the entire AI_Survival/ directory to the external HDD
3. Deploy (offline): plug in, run setup.sh, then start.sh or start-local.sh

Docker-mode settings are in .env:

Local-mode settings are in .env.local (created by install-local.sh). It uses the same variable names.

1. Download any GGUF model from Hugging Face
2. Place the .gguf file in the models/ directory
3. Switch to it: ./scripts/switch-model.sh <filename.gguf>

See models/README.md for more details.

Run the pre-deployment checklist:

./scripts/validate.sh

Checks project structure, configs, models, Docker images, and system requirements.

Q: Do I need internet to use this? No. Once the HDD is prepared (models downloaded, Docker images or local binaries saved), everything runs offline.

Q: Do I need Docker? No. Use ./scripts/setup.sh --local for local mode — only Python 3.11+ is required.

Q: What if Docker isn't installed on the target machine? Run ./scripts/download-docker.sh on an internet-connected machine to save the Docker installer to installers/docker/. The setup script will find it and tell you where to run it.

Q: How much RAM do I need? 4 GB minimum (TinyLlama only). 8 GB for 3B–7B models. 16 GB for comfortable 8B usage.

Q: Can I use my own models? Yes. Any GGUF model compatible with llama.cpp works. Drop it in models/ and switch to it.

Q: The server takes a long time to start. Large models (7B+) take 30–60 seconds to load on CPU. The start script waits up to 5 minutes.

• Docker: docker logs -f ai-survival-llama
• Local: cat data/logs/llama-server.log

Q: Port 3000 or 8080 is already in use. Change WEBUI_PORT or LLAMA_PORT in .env (Docker) or .env.local (Local).

Q: How do I update llama.cpp?

• Docker: edit LLAMA_CPP_VERSION in Dockerfile.llamacpp, rebuild, re-export
• Local: edit LLAMA_VERSION in install-local.sh, delete bin/llama-cpp/, re-run install-local.sh

AI_Survival/
├── Dockerfile.llamacpp          # llama.cpp server image (multi-stage build)
├── docker-compose.yml           # Docker orchestration
├── docker-compose.gpu.yml       # GPU override
├── .env                         # Docker-mode configuration
├── .env.local                   # Local-mode configuration (created by install-local.sh)
├── .dockerignore                # Build context exclusions
├── models/                      # GGUF model files
├── data/                        # Persistent data (WebUI DB, venv, logs, PIDs)
├── images/                      # Saved Docker image tars
├── installers/
│   ├── docker/                  # Docker installers for offline deployment
│   │   ├── windows/
│   │   ├── macos/
│   │   └── linux/
│   └── local/                   # Pre-cached binaries for local mode
│       ├── llama-cpp/
│       └── open-webui/
├── bin/                         # Extracted local binaries (created by install-local.sh)
├── config/                      # Server configuration reference
├── scripts/                     # All automation scripts
├── OFFLINE.md                   # Offline deployment guide
├── plan.md                      # Implementation plan
└── README.md                    # This file

This project assembles open-source components. See individual model licenses in models/README.md.

If you find this useful, consider supporting its development. Your contributions help keep the project maintained, fund new features, and cover infrastructure costs.