AgentShield

English | 简体中文

An AI security agent that actually launches the attacks itself. It starts by auditing and scoring your machine for privacy and vulnerabilities, then moves the red-vs-blue duel into a real Docker-isolated network: a Kali attacker box sends real packets, the WAF really blocks and really allows, and the outcome is judged by the target's own records. Runs 100% locally, MIT licensed.

Download video (2:42, Chinese narration and subtitles) · Subtitle file

A replay of an existing Docker drill: the same attack is blocked with the WAF enabled, succeeds with protection disabled, and is blocked again after protection is restored. The walkthrough explains the ReAct tool loop, human confirmation, isolation checks, and target-based judging. In live mode, one Agent coordinates the red-team tools and blue-team WAF.

The GIF above is a real run recording from an AgentShield live drill (model thinking pauses compressed): the red-team container attacks an admin login with SQL injection — 403-blocked while the WAF is on, normal traffic unaffected; after blue team mistakenly turns the WAF off, the very same payload obtains admin login credentials; the judge reads the target's own API to confirm loginAdminChallenge — model self-assessment is not trusted. When the drill ends, one click destroys every container, and all commands and outputs are persisted to disk.

An actual run through the Docker live-range entry: the Agent checks the existing range, proposes a command, receives need_confirm, and asks the user to confirm or cancel. The recording stops at confirmation; no attack was executed for this capture. Model waiting time is shortened.

This clip shows the fixed-flow rules simulation, with red/blue event feeds and before/after metrics. It uses synthetic events and sends no real attack packets; the Docker drill above demonstrates real execution.

User guide (Chinese) · Install & run

• Live drills (real attacks): spin up a training range inside a Docker-isolated network — a Kali container as red team, a toggleable WAF as blue team, OWASP Juice Shop as the vulnerable target. The agent walks you through the fight step by step: every attack action shows the full command first and runs only after you confirm; after blue team enables/disables protection, red team re-fires the same attack, so defense effectiveness is compared on the spot. See Live drills.
• Machine checkup: checks system security settings, optional LAN probing, outputs a 0–100 health score with per-item fix advice. System checks are currently mainly adapted for macOS.
• Skill auditing: local rules find security risks in Agent Skills; NVIDIA SkillSpector can be installed as a complementary review. Reports list findings, supporting evidence and scan scope.
• Simulated arena: rules mode needs neither Docker nor a model; agent mode requires models. Ten attack/defense scenarios (public Wi-Fi, malicious Skill, lateral movement, phishing & session theft, Web/API privilege escalation, supply-chain poisoning, AI agent prompt injection, guest device / office appliance access, cloud key & storage bucket, ops-agent banner compromise); red and blue models decide on their own, and tools only mutate the virtual scenario — no real packets. Tune the defense policy and compare both attack outcomes and normal-business impact. See arena scenario spec and live drill scenarios spec.
• Agent chat: hit AGENT at the bottom right — a ReAct loop acting step by step: read reports, audit skills, run commands (read-only whitelist runs directly, everything else asks you one by one), start the live range, delete reports (confirmation gate).
• Plain-language explanations: unfamiliar with security jargon? Read the model-generated explanations; want to dig deeper? Inspect the rules, evidence and raw reports.

NVIDIA review uses unmodified SkillSpector 2.12.0, with results shown separately from local checks. Install instructions: NVIDIA integration guide; completed verifications: integration log.

The regular arena is a simulation. Live drills move the duel into a Docker-isolated network where red team really fires packets and blue team really changes protection settings — the entry point is the "🔴 Docker Live Range" card on the arena page; click "Let the Agent guide me" to have the agent walk you through:

• Requirements: Docker (no Kali installation needed — the attacker box is the official Kali arm64 container image). The red-team container plus a deliberately vulnerable target (OWASP Juice Shop) run inside an internal network with no egress; attacks cannot reach your router or the internet.
• How it plays: the red-team model invokes real in-container tools (nmap/sqlmap/curl etc.); every attack action shows the full command and executes only after your confirmation. Blue team's moves are enabling/disabling the WAF in front of the target (red team cannot route around the WAF at the network level). Wins are judged by HTTP probes against the target, not model self-assessment. When a drill ends the scene stays up — browse logs and re-test as long as you like, then tear everything down with one click; every command and output is persisted.
• Named scenario scripts: four one-click drills with objective oracles — SQL injection session hijack (sqli_session), XSS encoded WAF bypass (xss_encoded_bypass), broken access-control user enumeration (bac_enumeration), SSH banner prompt-injection against an ops agent (ssh_banner_agent). The range card's scenario picker (or livelab.py run <scenario>) demos the attack, flips the WAF where relevant, re-fires, judges from the target's own records, and restores the WAF to block. See live drill scenarios spec.
• Be the red team yourself: don't want the model pulling the trigger? Run 08_arena/red_console.py — a guided console that lists attack vectors (SQLi / XSS bypass / broken access control / SSH banner poisoning), explains each step, shows the full command and fires inside the Kali box only after you confirm. One-off shots via livelab.py attack "cmd", or drop into the box with docker exec -it aslab-red bash. Or hand the whole attack to the agent: chat_cli.py --objective sqli|xss|bac|ssh|recon|custom runs an autonomous ReAct offensive loop — it picks its own tools (port recon, batch injection, single-shot HTTP, SSH banner), fires real packets, cross-checks the win with the judge probe and reports the spoils; a repeated action with no new info gets a nudge, the loop caps at 20 rounds. Default is confirm: every attack packet shows its full command before you nod. The arena page's Docker range card hosts the same console in the browser — direction picker, permission level, confirm box per strike, stop button. Need a target beyond the fixed range? --objective custom --declare 127.0.0.1:3000 (or the HOST:PORT field in the web console) declares your own host/LAN service: traffic goes through a single-purpose relay container that proxies only that one host:port — the Kali box stays air-gapped from everything else, with no judge on this line (evidence, not verdicts) and the channel auto-torn-down at wrap-up (livelab.py declare [HOST:PORT|clear]).
• Boundaries: network and web-application layers only. Wi-Fi RF scenarios (MITM, deauth) are out — Apple Silicon's built-in Wi-Fi has no monitor mode; that needs an external USB adapter plus a full Kali VM, a possible future option. The target ships with real vulnerabilities and must only ever run inside the isolated network, never exposed to any reachable network; red-team munitions are allowed only against in-range targets — if isolation fails, the lab refuses to start.
• Legality: only the targets inside your own containers may be engaged. Testing any system you don't own or lack written authorization for is illegal; this project neither provides nor assists such capability.

1. Start local Docker, configure and test a model connection in Settings, then open the arena.
2. Choose from the shared scenario list. SQL injection and XSS encoding-bypass Docker scenarios appear first.
3. Click “Let the Agent guide me”. The Agent checks or starts the range and plans each step from tool feedback. Review the complete command, then confirm, cancel, or enter your own response.
4. Refresh the range status to check containers and WAF mode. Use “Stop and clean up range” to finish; closing the page does not stop containers.
5. Expand the “🔴 Red Team Console · Launch an autonomous attack with the red agent” card and click Activate the red agent after choosing the direction, permission level, and extra requirements. In confirm mode each shot shows the full command first and waits for your nod in the box; when a round calls no tools, the loop ends and it wraps up with a report (ReAct convergence; step cap 20, adjustable 4–60 in the “max steps” field — hitting the cap forces a wrap-up and is marked in the report). Spoils are judged by target probes, not the model. Stoppable at any time; the scene is kept after.

There is one Agent guidance entry. The fixed-script run button has been removed; backend scripts remain for reproduction and regression checks. The synthetic rules demo needs no model; Agent guidance and red/blue model decisions require a model service.

When a confirmation question has no model-supplied choices, the UI supplies confirm/cancel buttons. Open questions have a text field and submit button. An unparseable Ask stops the turn instead of repeatedly retrying. After updating code and restarting the WebUI, reload the page and start a new conversation; sessions do not survive a server restart. Existing videos and GIFs show earlier recordings, so their button layout may differ from the current UI.

• Results depend on rule and test-case coverage; no findings does not mean no risk.
• Use test samples for drills — never production accounts or real secrets; use network checks only on devices you own or are authorized to test.
• Skill static scans and template reports work offline (--no-ai, --no-narrative); model-backed analysis needs the respective service.

Steps below use macOS; system checkup is mainly adapted for macOS. The main program uses only the Python standard library — the web UI, local rule scans and arena simulation need no extra Python dependencies and no DGX Spark.

Prepare Git and Python 3.11, and make sure python3 --version points to it. In your chosen directory:

git clone https://github.com/zhuangjia777/agent-shield.git
cd agent-shield
python3 -m venv .venv

If you already cloned it, just cd agent-shield — no need to clone again.

# First run: configure the LLM (copy config.json.example to config.json)
test -f config.json || cp config.json.example config.json  # keeps an existing config

Edit the three fields under cloud in config.json:

You can also fill this in via the web UI's Settings after startup. Rule scans and scripted demos need no model; red/blue agents, AI explanations and narrative reports do. Both arena sides inherit the main model by default, and each can be configured with its own endpoint, model and key in Settings. config.json is Git-ignored — never put real keys in the example config.

StepFun (阶跃星辰) is a supported cloud option: set base_url to https://api.stepfun.com/v1 and model to a current Step model ID (e.g. step-3.7-flash; see StepFun quick start), with the key created in the StepFun console.

Run inference on DGX Spark and connect AgentShield through a local OpenAI-compatible endpoint for agent guidance, red/blue decisions and report explanations. The model server and Docker range are separate processes; stopping a range does not unload the model.

NVIDIA DGX Spark has 128 GB of unified memory shared by the OS, weights, KV cache and Docker containers. Start with 27B, then try Flash-Next; load only one model at a time.

Drop-in alternative — ASUS Ascent GX10: an OEM build of the same NVIDIA GB10 Grace Blackwell platform — same 20-core Arm CPU (10× Cortex-X925 + 10× Cortex-A725), 1 PFLOP FP4 tensor performance, 128 GB LPDDR5x unified memory, and it ships NVIDIA DGX OS (ASUS tech specs). Everything in this section applies to the GX10 unchanged; only storage (1 TB / 2 TB / 4 TB options) and I/O details differ.

These are third-party Unsloth quantizations, not original Qwen weights. Flash-Next UD-Q4_K_XL is about 111 GB on disk, leaving little room on one Spark, so this example starts with a smaller quantization and 8K context. Validate capacity, quality and speed on your hardware; “Flash” is not a measured AgentShield speed claim.

1. Build llama.cpp with CUDA on Spark. First check that nvidia-smi and nvcc --version work with the installed NVIDIA driver and CUDA Toolkit. Run the following in Spark's Linux terminal; downloading source and weights requires network access. See the llama.cpp build instructions.

sudo apt-get update
sudo apt-get install -y git cmake build-essential libssl-dev libcurl4-openssl-dev
mkdir -p "$HOME/llm"
cd "$HOME/llm"
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4 --target llama-server
# Record the revision; use a recent version supporting the model architecture
git rev-parse HEAD

2. Start one model. Choose one command below, from the llama.cpp directory. The first launch downloads weights; wait for successful loading. Options are documented in the llama-server reference.

# Option A: Qwen3.8-27B
./build/bin/llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M \
  --alias agent-shield-local --host 127.0.0.1 --port 8000 \
  --api-key spark-local -ngl 99 -c 16384 -np 1 \
  --jinja --chat-template-kwargs '{"enable_thinking":false}'

# Option B: Qwen3.8-Flash-Next (stop option A with Ctrl+C first)
./build/bin/llama-server \
  -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL \
  --alias agent-shield-local --host 127.0.0.1 --port 8000 \
  --api-key spark-local -ngl 99 -c 8192 -np 1 \
  --jinja --chat-template-kwargs '{"enable_thinking":false}'

Non-thinking mode is an integration starting point so reasoning does not consume the limited output budget; this is not an optimal sampling recipe or capability benchmark. For unknown architecture/template errors, rebuild a llama.cpp version supporting the model. If memory runs out, reduce context or return to 27B. CPU offloading does not add physical memory on Spark's shared-memory system.

3. Check the endpoint and connect AgentShield. In another Spark terminal:

curl --fail http://127.0.0.1:8000/v1/models \
  -H 'Authorization: Bearer spark-local'
curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Authorization: Bearer spark-local' -H 'Content-Type: application/json' \
  -d '{"model":"agent-shield-local","messages":[{"role":"user","content":"Introduce yourself in one sentence."}],"max_tokens":256}'

Check for nonempty choices[0].message.content. Enter the following values in AgentShield Settings, or merge this cloud object into your existing config.json, keeping other configuration:

{
  "cloud": {
    "base_url": "http://127.0.0.1:8000/v1",
    "api_key": "spark-local",
    "model": "agent-shield-local",
    "max_tokens": 4096
  }
}

cloud is the project's configuration name for OpenAI-compatible endpoints; inference here remains local to Spark. Do not leave the key empty: the current client requires a nonempty key to enable this path. The example key matches the server flag and is for loopback use. OPENAI_BASE_URL, OPENAI_API_KEY and OPENAI_MODEL, when set, override the main model's file configuration. Let both arena roles inherit the main model: they share the server but keep separate conversation contexts.

If AgentShield runs on a Mac/PC and the model runs on Spark, keep this SSH tunnel running on the Mac/PC, replacing the user and hostname:

ssh -N -L 8000:127.0.0.1:8000 spark-user@spark-host

Keep AgentShield's URL as http://127.0.0.1:8000/v1; the model port need not be public. Start AgentShield using the quick start, then test the model connection in Settings, an agent conversation and a red/blue drill. When Spark only supplies inference, system checks and Docker tools still execute on the AgentShield host. System checks currently primarily target macOS.

To stop: press Ctrl+C in the model terminal to release model memory, Ctrl+C in the tunnel terminal to close forwarding, and use “Stop and clean up range” for Docker. For hardware validation, record the llama.cpp commit, quantization revision, context length, time to first token, generation speed and peak memory.

In the project directory:

.venv/bin/python 04_web/app.py --open

Your browser opens AgentShield. Keep the terminal running while in use; Ctrl+C stops the server. Next time, just run the same command from the project directory — or use the one-liner .venv/bin/python 08_arena/livelab.py web (runs from any directory; --bg detaches; a busy port is reported instead of silently stacking a second instance).

If port 8787 is taken:

.venv/bin/python 04_web/app.py --port 8788 --open

Press Control + C (Ctrl+C) in the terminal that started it. Closing the browser tab alone does not quit the program.

If you can't find that terminal, see which process holds the default port:

lsof -nP -iTCP:8787 -sTCP:LISTEN

Once confirmed it's AgentShield:

lsof -tiTCP:8787 -sTCP:LISTEN | xargs kill

If you started on another port, replace 8787 accordingly. Run the start command again next time.

Install this only if you want the NVIDIA Skill security review. It lives in its own Python environment; the commands below require uv and run from the project directory:

uv venv --python 3.14 .venv-skillspector
uv pip install --python .venv-skillspector/bin/python \
  -r 09_integrations/nvidia/requirements.lock
.venv/bin/python 09_integrations/nvidia/record_install.py

Installation downloads dependencies over the network. The last command verifies component contents and records the install path. Afterwards open NVIDIA Skill review. More details: integration install & reproduction guide.

All commands run from the project directory:

# Evaluate a Skill with local rules only, no model
.venv/bin/python 02_scan/cmd_scan.py --skill 06_samples/vulnerable-skill --no-ai

# Generate JSON/Markdown/HTML reports with model-written narratives
.venv/bin/python 03_ai/report.py --input reports/eval_vulnerable-skill/results.json

# Offline report with template narratives
.venv/bin/python 03_ai/report.py --input reports/eval_vulnerable-skill/results.json --no-narrative

# Run the benchmark suite
.venv/bin/python tests/run_evals.py

The live range can also be driven without the web UI:

.venv/bin/python 08_arena/livelab.py start   # bring up the range (runs network-isolation checks)
.venv/bin/python 08_arena/livelab.py status  # container & WAF status
.venv/bin/python 08_arena/livelab.py scenario            # list the named scenario scripts
.venv/bin/python 08_arena/livelab.py run sqli_session    # demo + judge one scenario
.venv/bin/python 08_arena/livelab.py attack "nmap -sT -Pn -p 8080 aslab-blue | grep open"   # be the red team yourself
.venv/bin/python 08_arena/livelab.py stop    # tear everything down

# drive the agent from a terminal, no browser ( -m observer|confirm|auto )
.venv/bin/python 03_ai/chat_cli.py
.venv/bin/python 03_ai/chat_cli.py -m auto "run the sqli_session scenario and report the verdict"

# guided red-team console: pick an attack vector and target, confirm each shot
.venv/bin/python 08_arena/red_console.py

# autonomous offensive loop: the agent picks its own tools and fires until it stops calling them
.venv/bin/python 03_ai/chat_cli.py --objective sqli
.venv/bin/python 03_ai/chat_cli.py --objective ssh -m auto

Commands passed to attack run inside the Kali container for real (nmap/sqlmap/curl/ssh available) under the same target allowlist, defense-in-depth blocklist and audit trail as the agent; targets are restricted to aslab-blue:8080 (traffic must cross the WAF) and aslab-ops (the SSH line) — anything external is refused. Blue-team toggle: livelab.py waf block|bypass; win check: livelab.py judge. red_console.py wraps the same channel interactively: it lists attack vectors (SQLi / XSS bypass / broken access control / SSH banner poisoning), explains each step, shows the full command and fires only after you confirm.

Open “Live console” in the Docker range card to switch between WAF logs, target logs and Agent attack output. It refreshes every two seconds and supports pause, auto-scroll and copy. Polling stops while collapsed or the page is hidden. This is read-only; attack output appears after commands finish, and history may be truncated.

Start the range first, then run docker ps in a host terminal to check these containers:

Open an interactive shell in the Kali container:

docker exec -it aslab-red bash

Show the last 100 log lines and follow new output from the WAF or target (use separate terminals):

docker logs -f --tail 100 aslab-blue
docker logs -f --tail 100 aslab-target

In Docker Desktop, select the container under Containers, then open Exec for a terminal or Logs for service output.

docker exec opens a new shell; it does not replay the Agent's previous commands. The Agent's attack commands and outputs are recorded in logs/arena_live/events.jsonl under the project directory. Container logs primarily contain service output.

Type exit to leave the shell, or press Ctrl+C to stop following logs. Neither stops the container. If the container is missing or stopped, start the range first. To finish the drill, use “Stop and clean up range” in the WebUI.

The local health score starts at 100 and loses points per finding: −25 critical, −12 high, −5 medium, −2 low, floor 0. The score is computed by rules; the model explains it but never changes it.

Each finding's risk value is computed as below, then bucketed into a severity:

risk = impact × exploitability × evidence_confidence × exposure

Impact and exploitability range 1–5 (higher = easier to exploit); evidence confidence is 0.5, 0.75 or 1.0; exposure is 0.5, 1.0 or 1.5. See findings.py for the exact math.

NVIDIA SkillSpector scores go the other way — higher means riskier — and never add to the local health score. With model analysis enabled it may affect NVIDIA's risk score.

Local rules currently hit all expected checks on the four sample Skills (5 planted checks in the vulnerable sample; the ops-agent sample covers instruction-trust, plaintext key, eval-exec and supply-chain pipe), with no high-severity false positives on the benign and hardened samples. This covers those samples only — not all 30 evaluation cases. Details: BENCHMARK.md. The NVIDIA engine's coverage and runtime are logged separately in the integration log.

The arena page uses red/blue agents by default: each side has its own context, the model picks tools from observations and keeps acting on results. Red team can observe and attempt targets; blue team can view alerts, enable protections and check business traffic. Up to 4 decisions per side; tools only mutate the virtual scenario, and the judge is rule-based. A no-model scripted demo remains available.

• Provenance verification is wired into Skill review: the official OMS verifier with pinned NVIDIA trust certificates, showing unsigned / failed / passed separately. Passing verification still requires security checks.
• The OpenShell execution adapter provides scoped tools, strict file/network policies and operation logs. Real isolated execution needs an available gateway; gateway testing is not yet complete.
• SkillEvaluator Tier 3 uses the official tool with four with/without-Skill task pairs. Data-format validation passes; effectiveness conclusions must come from complete experiment results on both groups; failed runs never produce a lift score.

Install, run and output locations: integration guide.

Next steps, roughly in order:

1. More rule-arena scenarios — RAG knowledge-base poisoning and container privilege-boundary scenarios, extending the existing declarative scenario framework.
2. More live drill scripts — an nmap reconnaissance script and a sqlmap automation script as named one-click scenarios alongside the current three.
3. OpenShell gateway testing — complete the isolated-execution gateway so dynamic Skill review runs against real sandboxes.
4. Adaptive presentation — automatic recognition of the user's expertise level instead of the current manual depth toggle.

01_specs/        architecture, feature specs, verification logs
02_scan/         scan entry, system & network checks, scoring rules
03_ai/           model calls, result explanations, report generation
04_web/          local web UI
05_skill_eval/   Skill check rules & NVIDIA scan entry
06_samples/      vulnerable / ops-agent / hardened / benign sample Skills
07_evals/        evaluation cases and expected results
08_arena/        arena simulation + live range control (livelab.py, waf.py)
09_integrations/ NVIDIA component versions, locks, runtime adapters
10_skills/       AgentShield audit-skill drafts (effect comparison pending)
docs/assets/     README cover and doc images
reports/         locally generated reports, not committed
tests/           automated checks and evaluation scripts
config.json      model endpoints, keys, Ollama config, not committed
BENCHMARK.md     test results and pending evaluation items

MIT, see LICENSE.