AISee

• What Is AISee?
• Installation
  • Prerequisites
  • Steps
• Quick Start
• Installing Models
  • Built-In Catalog
  • Other Models
• REST API
  • Server
  • Reference
  • Authentication
• MCP Server
• Agent Files
• Credentials
• Local Data
• Troubleshooting
• License

AISee is a tool that gives AI agents eyes and ears. It serves vision-language and audio (ASR + diarization) models in docker containers on a GPU host and answers questions about images, video, and audio files, over a CLI, a REST API, or an MCP server.

There are five kinds of queries:

• look - free-form question, returns text. OCR, descriptions, "where is the button".
• assert - an expectation to verify, returns {pass, reason, evidence}. Meant for visual regression and e2e checks; the CLI exit code follows the verdict.
• watch - whole-video analysis, chunk by chunk, at a chosen fps. Given an expectation it returns per-chunk verdicts and failing_ranges (the time spans where it broke); given a question it returns per-chunk notes and a synthesized answer for the whole video.
• transcribe - word-timestamped transcript of EVERY audio lane of a recording (an audio file, or a video container's audio); diarize: true (CLI --diarize) adds per-lane speaker attribution. A lane is one audio track or one channel of a stereo/multi-channel track (stereo often encodes two separate feeds); AISee never interprets or merges lanes - that is the consumer's job - and bit-identical lanes (equal decoded PCM) are detected and processed once. Rendered per-lane transcripts (.txt/.srt/.vtt + full JSON, plus RTTM when diarization was on) are downloadable artifacts - one by one, or zipped via aisee task download (which also saves the full task as results-<id>.json, for every task kind).
• diarize - who spoke when: speaker turns with timestamps per lane, no transcript. Downloads the same way (diarize-<id>.zip: per-lane RTTM + JSON).

The CLI is a thin client of the REST API, so there is a single code path: anything the CLI does can be done with curl, and both feed the same task queue. All queries are asynchronous - you get a task id back and poll it; progress like "model is loading" or "chunk 3/12" is visible per task.

• Linux GPU host with an NVIDIA GPU
• docker + NVIDIA Container Toolkit
• ffmpeg (includes ffprobe; apt install ffmpeg)
• Python 3.12+, uv
• HuggingFace token for gated models
• NGC API key if serving images come from nvcr.io

Get the source onto the host (git clone or rsync), then:

cd ~/aisee
uv sync                        # .venv from uv.lock, aisee installed editable

./aisee install                # checks docker/nvidia/ffmpeg, creates ~/.aisee
./aisee creds set HF_TOKEN     # hidden prompt; lands in ~/.aisee/credentials.json
./aisee creds set NGC_API_KEY  # only if you pull nvcr.io images

./aisee at the repo root runs the venv'd CLI from anywhere, and bootstraps .venv itself on first use, so the explicit uv sync is optional. Symlink it into ~/.local/bin if you want it on PATH.

./aisee uninstall removes everything AISee put on the host; --keep-cache preserves the downloaded weights.

./aisee model install qwen3-vl-30b-a3b-instruct
./aisee assert shot.png -e "the Start button is visible and enabled"

The first query starts the API daemon and the model container by itself. Weights download on first load - the task sits in model_loading while that happens, which can be a while for the big models. The first installed model becomes the default.

More examples:

./aisee look shot.png -q "What error message is shown?"
./aisee look page.png -q "Where is the search box?" --model holo1-5-7b
./aisee assert run.mp4 -e "the app launches into the main menu" --native
./aisee watch run.mp4 -e "the frame counter increases monotonically" --fps 8
./aisee watch run.mp4 -q "describe what the user does" --fps 2
./aisee transcribe meeting.mp4                      # transcript only (every audio lane)
./aisee transcribe meeting.mp4 --diarize --max-speakers 5   # + speakers per lane
./aisee diarize meeting.wav
./aisee status

Video can be sent as sampled frames (--frames N spread evenly, or --fps R) or as the video itself (--native, for models that support it). --context "..." / --context-file f.txt passes background the model can't see in the pixels ("the left panel is the scene tree"). --no-wait prints the task id and returns; aisee task show <id> polls it later.

./aisee model install <catalog-slug or HF-id> [--gpu-frac F --image I --port P --idle-timeout S --arg X]
./aisee model list | start <slug> | stop <slug> | logs <slug> | default <slug> | remove <slug>

install only writes a registry entry (~/.aisee/models/<slug>.toml) and picks a port; the download happens on first start. stop frees the GPU but keeps weights and config. remove drops the entry; weights stay in the shared cache.

The built-in catalog covers eight vision models and two audio models, measured on a DGX Spark GB10 (2026-07 / 2026-08). Installing by slug applies the serving flags each one needs:

Each catalog model states an absolute GPU-memory requirement in GiB (weights + ~4 GiB runtime + a KV-cache pool sized for ~2.5 full contexts where affordable), so the sizing is portable across GPUs. At model install time the requirement is adapted to the detected GPU: the serving fraction becomes mem_gib / GPU memory (capped at 0.97 on discrete cards and 0.92 on unified-memory systems, where the GPU pool is also system RAM and the reserve keeps the OS and the small audio models alive), and the context window is the largest size - up to the checkpoint's native limit - whose KV cache fits inside that slice. The Qwen3-VL and Cosmos families serve with an fp8 KV cache (halves KV cost; validated with exact-OCR and deep needle-retrieval tests with no quality loss), which is what makes their native 256k contexts affordable. On the known tiers: GB10 (~120 GiB unified) serves the whole catalog at 256k (128k for Nemotron/Holo/UI-TARS - their checkpoints' native limit); a 96 GB discrete card serves everything at 256k except the two dense-KV 32B-class models (qwen3-vl-32b, cosmos3-super - 128k there); a 48 GB card fits the 7-17 GiB models, while the two big Qwens (~62 GiB weights) do not fit at all (install warns). A model start is refused up front - with a GiB message - when the requirement does not fit next to the already-running models (plus a system reserve), or when the GPU's actually-free memory says otherwise (measured margins: a large load must leave 10 GiB free on unified hosts, audio engines 3 GiB, discrete GPUs 2 GiB). On unified hosts a large model is also refused while audio jobs are in flight - the cold load would starve them; retry when they finish. Media budgets: max_images is sized per model so a full batch of 1080p stills fills the context (~2-3.3k tokens per still depending on the preprocessor - e.g. 60 for the Qwen3/Cosmos family at 128k, 46 for Holo/UI-TARS, 36 for Nemotron); video is 1 per request, sampled up to the model's frame budget (default 96 - a cap, not a quota: short clips cost only the frames they contain; each sampled frame keeps ~720p detail at any cap, ~515 tokens/frame, so cost grows linearly with sampled frames). Execution mode is also per-GPU: unified-memory systems serve with --enforce-eager (CUDA graphs measured slower there), discrete GPUs keep CUDA graphs (3-4x faster). Each model runs up to concurrency inferences in parallel (default 3; vLLM batches them) - concurrent bursts gain ~1.4-2x and watch chunks are processed in parallel. Context length is the expensive knob - vLLM reserves KV-cache memory for the full max_model_len inside the model's gpu_frac slice, so raising it costs GPU memory even for short requests; override with --max-model-len / --gpu-frac at install.

Audio models are first-class citizens of the same lifecycle: install/remove/start/stop/ logs/default, the same registry TOMLs, idle-timeout unload, and auto-load on first use all work identically. They differ in serving: instead of vLLM they run small FastAPI apps in locally built images (aisee/audio-nemo, aisee/audio-pyannote; built automatically from res/serving/ on the first start, ~10-20 min one-time), sized honestly (7 + 4 GiB) so they co-reside comfortably next to a vision model. Both containers GPU-gate at startup: if inference is not actually running on CUDA (the classic ARM64 failure mode is a silent CPU fallback), the container exits loudly instead of serving 30-120x slower. The first model installed for a capability becomes that capability's default (defaults.default_transcribe_model / default_diarize_model in config.toml). The pyannote weights are HF-gated: the host's HF_TOKEN account must have accepted the license forms on pyannote/speaker-diarization-3.1, pyannote/segmentation-3.0, and pyannote/wespeaker-voxceleb-resnet34-LM.

Beyond the catalog, any model works that the serving image's vLLM can run as a multimodal chat model - i.e. it accepts OpenAI-style chat/completions with image_url content parts. In practice that is the Qwen-VL family and its derivatives (Holo, UI-TARS), InternVL, Pixtral, Gemma 3, LLaVA-style models, NVIDIA's Cosmos/Nemotron VL models, and most other open VLMs - see the vLLM supported-models list for the image in use.

Install by HF id and pass whatever serving flags the model needs:

./aisee model install org/Model --gpu-frac 0.3 --arg --enforce-eager --arg --trust-remote-code

Things to know when going off-catalog:

• Video: --native and watch need vLLM video_url support for that architecture (Qwen-VL, Cosmos, Nemotron-VL have it; grounding-tuned 7Bs like Holo/UI-TARS read a clip as a single frame). Frame-based queries (--frames/--fps) work with any image-capable model.
• Reasoning models (answers arrive in reasoning_content): handled automatically; give them --max-tokens 2048.
• Quantized checkpoints (NVFP4/FP8/AWQ): vLLM auto-detects the quantization from the checkpoint - don't pass --quantization.
• Custom code models: add --arg --trust-remote-code.
• Different serving image: --image swaps the container image per model (e.g. an architecture only supported by a newer vLLM or a vendor build); nvcr.io images need the NGC key.
• Per-request budgets: max_images sized so 1080p stills fill the context (install default 16; see the catalog notes), 1 video (24 server-sampled frames, keeping each frame at ~720p); AISee's frame sampling respects them. There is no hard video-length limit - only temporal resolution (the frame budget spread over the clip); use watch for long videos.

Several models can be installed at once, and models co-reside whenever their GiB requirements fit together (the small audio pair, ~11 GiB, runs comfortably next to the default VLM; two large VLMs generally do not fit). A start that would not fit next to the running models - by the models' stated GiB requirements plus a system reserve, or by the GPU's actually-free memory - is refused up front with a clear GiB-denominated error (HTTP 409) instead of letting the container crash-loop. To co-locate more, lower --gpu-frac / --max-model-len per model at install. Tasks queue FIFO per model, with up to concurrency running at once.

A model idle longer than its idle_timeout (default 900 s, 0 disables) is stopped automatically to free the GPU. The next query targeting it starts it again; the task reports model_loading in the meantime.

./aisee api start [--port N] [--host 0.0.0.0|127.0.0.1]
./aisee api stop | status

--port and --host persist to ~/.aisee/config.toml. 0.0.0.0 (the default) serves the LAN, 127.0.0.1 is local-only. The CLI starts the daemon on demand; --no-autostart disables that.

api start spawns a one-off daemon that dies with the host. For a server that survives reboots, run the same entry point under systemd. Model containers already come back on their own (--restart unless-stopped), so only the API needs a unit. Preferred - a system unit (requires sudo; picks up the user's docker group membership at start):

# /etc/systemd/system/aisee-api.service
[Unit]
Description=AISee REST API server
After=network-online.target docker.service
Wants=network-online.target

[Service]
Type=simple
User=<user>
ExecStart=/home/<user>/aisee/.venv/bin/python -P -m aisee.server
WorkingDirectory=/home/<user>/.aisee
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now aisee-api

Without sudo, a user unit (~/.config/systemd/user/aisee-api.service, same [Service] minus User=, WantedBy=default.target) plus loginctl enable-linger <user> works - with one caveat: the systemd user manager freezes its group list when it starts, so if the user was added to the docker group after that, the API cannot reach the docker socket until the manager restarts (symptom: every model reports installed while its container runs; check with systemd-run --user --pipe id). The system unit does not have this problem.

Once under systemd, ./aisee api stop && ./aisee api start silently does nothing (stop finds no pidfile because systemd owns the process; start sees the healthy API and returns) - restart with systemctl restart aisee-api (or systemctl --user restart aisee-api) instead. Logs move to journalctl -u aisee-api.

The CLI also works from other machines - --server http://HOST:PORT or export AISEE_SERVER=http://HOST:PORT - media files are uploaded with the request.

Everything is under /v1; OpenAPI schema at /openapi.json.

curl -s -X POST http://HOST:PORT/v1/tasks \
  -F files=@screenshot.png \
  -F 'params={"kind":"assert","expectation":"the Start button is visible"}'
# {"id":"3f2a..."}
curl -s http://HOST:PORT/v1/tasks/3f2a...

A task moves through queued, preparing_media, model_loading (only when cold), running, and ends done, failed, or canceled. timings breaks out model_load_s, media_prep_s, inference_s, and (once finished) total_s - wall-clock from submission to the terminal state, also shown in the console's tasks table.

Uploads are deduplicated: media is stored content-addressed by the SHA-256 of the file bytes (kept for blob_ttl_hours, default 24 h; reuse refreshes it), and a media entry can be "sha256:<hash>" instead of a file - probe with GET /v1/blobs/{sha256} first (hash via sha256sum / shasum -a 256 / python hashlib). The CLI and the web console negotiate this automatically, so re-submitting the same video skips the upload; POST /v1/blobs uploads media without creating a task (this is also how remote MCP clients get local files to the server).

Auth is off by default. Two optional bearer tokens split access into roles:

• AISEE_API_TOKEN (consumer): guards the query/read endpoints - submitting and reading tasks, listing models/catalog/GPU stats. Hand this one to agents and users of the service.
• AISEE_ADMIN_TOKEN (admin): guards model management (install/uninstall/start/stop). Accepted on consumer endpoints too, so an admin needs only one token.

With only the consumer token set, it guards everything (single-token mode). With both set, a consumer token on an admin endpoint gets 403; a missing or wrong token gets 401. The console (/), /v1/describe, and /v1/health are always open.

./aisee creds set AISEE_API_TOKEN     # consumer token
./aisee creds set AISEE_ADMIN_TOKEN   # admin token

Tokens set through creds set apply immediately (the store is read per request); tokens set as environment variables of the daemon require a restart. The CLI picks tokens up from env or the creds store on its own (admin preferred when present; --token overrides). Unset the credentials to go open again.

The API server also speaks MCP (Model Context Protocol, streamable HTTP) at http://HOST:PORT/mcp - nothing to install on the client, point an agent harness (Claude Code, Cursor, etc.) at the URL. It exposes AISee as native tools: look, assert_visual, watch, transcribe, diarize, list_models, list_tasks, get_task, cancel_task, describe, health. It is a thin adapter over the same REST API and intentionally carries consumer capabilities only - model management is not reachable over MCP.

Register it in an MCP client config:

{
  "mcpServers": {
    "aisee": {
      "type": "http",
      "url": "http://HOST:PORT/mcp",
      "headers": { "Authorization": "Bearer <consumer token>" }
    }
  }
}

The headers entry is only needed when auth is enabled; /mcp is guarded like any consumer endpoint. Query tools block until the result is ready (a cold model can take minutes); watch accepts wait=false to return a task id for polling with get_task. Media paths are resolved on the AISee host - the files must already exist there (transfer them first, or use the REST API, which uploads).

Two ready-made role instructions for AI agents live at the repo root:

• aisee.consumer.agent.md - for agents that use AISee to see: query kinds, CLI/REST/MCP access, behavior to plan around, limitations.
• aisee.admin.agent.md - for agents that operate an AISee host: installing AISee locally or on a remote machine, tokens, model and server management, troubleshooting.

There is no credentials file in the repo. Lookup order: environment variable, CLI parameter, ~/.aisee/credentials.json, interactive prompt (hidden input, offers to save). Manage the store with ./aisee creds set|unset|list; list masks values. Keys used: HF_TOKEN, NGC_API_KEY, AISEE_API_TOKEN. The server never prompts - a task that needs a missing credential fails with a hint instead.

Everything AISee puts on the host, by who creates it:

uv sync (or the launcher's first run) creates .venv/ inside the source checkout - the Python environment with aisee and its dependencies (fastapi, uvicorn, httpx, pydantic, python-multipart, mcp). Nothing outside the checkout.

./aisee install creates the state directory ~/.aisee/ (override with AISEE_HOME):

~/.aisee/
  config.toml          # api host/port, defaults (fps, idle_timeout, task retention, blob TTL)
  credentials.json     # HF/NGC/API tokens, 0600; written by `creds set` or prompts
  models/<slug>.toml   # per-model serving config: image, port, mem_gib/gpu_frac, vllm args
  hf-cache/            # shared model-weights cache, mounted into every container;
                       #   by far the biggest item (tens of GB per model)
  tasks/tasks.db       # sqlite task store (statuses, progress, timings, results)
  tasks/blobs/         # content-addressed uploads (sha256-named; upload dedup); GC'd
                       #   after blob_ttl_hours (default 24), refreshed on reuse
  tasks/media/<id>/    # per-task media (hardlinks into blobs/) + derived frames/chunks;
                       #   GC'd with the task after task_ttl_hours (default 24)
  logs/api.log         # API daemon log
  run/api.pid          # daemon pidfile

It installs nothing system-wide: docker, the NVIDIA toolkit, and ffmpeg are prerequisites it only checks for.

./aisee model start (or the first query) pulls the model's serving image into the docker image store (nvcr.io/nvidia/vllm is ~20 GB, shared by most catalog models) and runs one container per model, named aisee-<slug>. Weights download into hf-cache/ on the first load.

./aisee uninstall stops and removes all aisee-* containers and deletes ~/.aisee/ (--keep-cache spares the weights). It does not touch the docker images, the source checkout, or .venv - remove those by hand (docker rmi ..., rm -rf ~/aisee) if you want a clean host.

• Task sits in model_loading: normal on first use (weights download, tens of GB) and after an idle unload (about 2 min to reload). ./aisee model logs <slug> tails the vLLM log.
• failed with "Free memory ... is less than desired": something else holds the GPU. Stop it (./aisee model stop <slug>, or whatever non-AISee container is running) and retry.
• HF 403 on a gated model: the token's account must accept the license on the model page first.
• failed to discover GPU vendor from CDI (docker cannot see the GPU): the NVIDIA Container Toolkit is missing or unconfigured - install it, then sudo nvidia-ctk runtime configure --runtime=docker && sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml && sudo systemctl restart docker. ./aisee install checks for this.
• Reasoning models (cosmos family) answering empty or unparseable: handled by a fallback, but give them room - --max-tokens 2048.
• "Could not open video stream" server-side: shouldn't happen through AISee (video is re-encoded to MJPEG-AVI exactly because serving containers often lack H.264), but raw H.264 sent by hand will do this.
• After updating the source: uv sync, then restart the API - ./aisee api stop && ./aisee api start, or systemctl restart aisee-api on a persistent (systemd) install, where the stop/start pair silently no-ops. A running daemon keeps executing the old code.

Apache 2.0