NEW
LogLM and open-weights model results now in Detection →
COMING NEXT
Harness for Vulnerability Detection and Patching. See roadmap →
SOCBench
An Open Benchmark for AI in Cybersecurity Operations.
What you can't measure, you can't improve. What you can't measure, you
also can't validate. AI in cybersecurity operations is outpacing both.
Vendors ship new agents faster than anyone can compare them on telemetry
that resembles what defenders actually see.
SOCBench is being built to be that benchmark: exhaustive and
open, putting AI systems through the actual work a SOC does (detection,
triage, investigation, hunting, detection engineering, threat
intelligence) and scoring them on the dimensions that decide whether
anyone can ship them.
Today, one capability is published: Detection. 3 frontier LLMs
× 4 analyst personas × 1,205 network-flow units, scored against ground
truth. The rest of the SOC follows.
SOCBench grew out of work that DeepTempo does to validate and improve
their LogLM and the open source AI-SOC Vigil. Everyone is welcome to
contribute and to help lead the SOCBench project.
The Benchmark
SOCBench scores AI systems on four dimensions: efficacy, cost, latency, and
reliability. All four count.
Efficacy
Does it get the right answer?
F1precisionrecallverdict accuracy
Cost
What does it actually take to run?
USD / alerttoken economics
Latency
How fast does it respond?
MTTDP50P95
Reliability
Does it return well-formed output, every time, within budget?
completion ratedefect rate
Current Capabilities and Road Map
Detection
Live
Classify flows, logs, or alerts as malicious or benign (the core work of the monitoring tier).
Live = published results ·
Roadmap = scoped, not yet measured ·
Exploring = open question whether a clean benchmark is possible
Capability · Live
Detection
The Detection benchmark scores three lanes on 1,205 evaluation units:
Frontier LLMs. Claude Opus 4.7, GPT-5.4, Gemini 2.5 Pro.
Open weights. Foundation-Sec-8B, Seneca-QwQ-32B, GLM 5.2.
Foundation model for cybersecurity. LogLM, DeepTempo's encoder-only model.
All LLMs run in the same detection harness: four analyst personas with network-flow tools submitting per-unit verdicts.
LogLM scores sequences directly, projected onto the eval-unit class prior.
System
Lane
Verdict F1
Per-flow F1
Note
loglm
deeptempo · projected
Purpose-built
0.95
0.99
overall winner · <$0.0001/alert
claude-opus-4-7
anthropic
Frontier LLM
0.93
0.65
best LLM · soc_analyst
gemini-2.5-pro
gemini
Frontier LLM
0.88
0.52
most balanced · soc_analyst
gpt-5.4
openai
Frontier LLM
0.86
0.44
best ops · threat_analyst
sec-8b
open weights
Open weights
0.75
0.37
threat_analyst
seneca-qwq-32b
open weights
Open weights
0.73
0.46
soc_analyst
glm-5.2
open weights
Open weights
0.48
0.55
22% coveragesoc_analyst
Cost, reliability, and false positives
Overall across all 1,205 units. LLM bars are the mean across 4 personas; LogLM is the single classifier. The per-flow / verdict F1 chart further down responds to the split tabs.
Avg. cost per alert
$ per verdict, log scale.
FP rate
Benign units wrongly flagged malicious.
Completion rate
Renderings completed within budget.
* glm-5.2 reached only 27% of the 1,205 shared units (328 of 1,205); bars use that subset and may not be representative of the model at full coverage.
benign: every flow in the evaluation unit was benign. In the chart and table below, Verdict acc is the true-negative rate (specificity).
per-flow F1 and verdict F1, by split
Mean across the four personas. Hover bars for exact values; click a legend entry to toggle a series.
Detection · How it's measured
Methodology
SOCBench starts with a labeled NetFlow corpus that is provided here. Similar approaches on other logs are planned. From these logs a per-provider score is calculated in seven deterministic steps, enabling easy comparisons. Four analyst personas with different tool budgets are defined and work the same evaluation units; submissions are scored against held-out gold labels.
Each persona has its own turn / tool-call / wall-clock budget and a defined toolset. Higher-tier personas inherit everything below them. The SOC analyst is the lean baseline; the detection engineer carries the full toolkit.
Tool
soc_analyst
threat_analyst
adversary_hunter
detection_engineer
list_pairs
●
●
●
●
get_pair_timeline
●
●
●
●
get_flows
●
●
●
●
host_rollup
●
●
●
●
submit_assessment
●
●
●
●
top_destinations
·
●
●
●
pair_stats
·
●
●
●
port_proto_matrix
·
·
●
●
rarity_stats
·
·
●
●
Budget(turns / calls / wall)
4 / 6 / 60s
8 / 12 / 120s
10 / 16 / 150s
12 / 20 / 180s
Models in this run:anthropic = claude-opus-4-7gemini = gemini-2.5-proopenai = gpt-5.4.
OpenAI receives a 1.5× persona-budget multiplier (its reasoning chains are chattier; without it, large host_egress units exhaust budgets before submission). Disclosed for fair comparison.
More detail
Blogs and docs explaining the methodology in more depth, including the corpus breakdown, eval-unit construction, scoring-lens math, run-metadata schema, and deterministic-replay tuple, are coming soon. If you look at the underlying GitHub you can see some of this explained in /docs and in the readme. This simple website is by design higher level.
Detection · Per-persona detail
F1, cost, and reliability per persona
All 25 configurations plotted on the F1 / cost frontier: 6 LLMs × 4 personas, plus LogLM. Winners on each contract dimension called out below (LLMs only, since LogLM has no personas).
At one million alerts a day, the cheapest LLM stack ($0.057 per alert) is roughly $60,000 of inference every day, before a single analyst looks at the output. At telco scale (a large telco active in the Vigil community reports approximately 70 million flow events per minute, which batches into roughly 2 billion alerts per day at 50 flows per alert), the LLM bill runs to hundreds of millions of dollars per day ($114M at the cheapest provider, ~$302M at the priciest).
F1 vs cost per alert, per persona
Cost is dollars per alert (one evaluation unit). Up and to the left is better. Color = provider (legend in chart). Shape = persona (legend below). Hover a point for the exact values; click a provider in the legend to toggle.
For each contract dimension, the winning (LLM × persona) configuration across all 24 combinations. LogLM excluded here since it has no personas.
Best efficacy
0.93
verdict F1
0.65
per-flow F1
anthropic · soc_analyst
Lowest cost
$0.006
per alert (total $7.10 / 1,205)
sec-8b · soc_analyst
Most reliable
99.3%
completion rate
openai · detection_engineer
All 25 rows (24 LLM configurations plus LogLM)
Run it yourself
Reproduce
SOCBench is open. Clone, set provider keys, run the launch entry point. Or skip the run and re-score the published artifacts directly. Everything lives in the repo.
A full run (~1,500 units × 4 personas × 3 providers) costs roughly $1,300 to $1,500 USD in LLM spend at the pricing snapshot dates in this report. A smoke run (8 units) is under $10.