Aria AI Security Research

Your SOC Agent's Final Answer Is Lying to You

Why security agents must be evaluated on their investigation trajectory — not their verdict — and what 120 fully-instrumented SOC episodes taught us about the gap between calling tools and actually understanding an incident.

TL;DR. A TF-IDF + logistic-regression baseline scores a perfect 1.00 macro-F1 on alert severity classification. That does not make it a competent analyst. Final-answer metrics saturate quickly. SOC-AgentBench grades the whole investigation. A 1.5B agent that succeeds at 88% of tool calls still finds the correct root cause only 58% of the time.

1. A perfect score that means nothing

While building baselines we hit a number that should make anyone suspicious: a bag-of-words classifier reached 1.00 macro-F1 on severity classification on the test split (eval_results.json). Severity labels correlate with surface vocabulary. A SOC analyst's job is to investigate: pull the right logs, build a timeline, distinguish true positives from noise, and commit to a root cause without inventing evidence or isolating a production host "just to be safe".

Hugging Face's Is it agentic enough? argument applies hardest in security: the path is the product.

2. Why security agents need trajectory evaluation

The verdict is low-entropy; the investigation is not. Priors can fake verdict accuracy.
Hallucinated evidence is worse than no evidence. Fabricated log lines poison response.
Actions have blast radius. Tool-poisoning success rates above 60% make forbidden-action scoring mandatory.

3. Anatomy of an evaluable investigation

SOC-AgentBench packages 120 episodes across 12 attack scenarios and 9 log sources. Every episode carries the world and the answer key: gold evidence, gold timeline, and forbidden actions. Seven tools run against episode-local data only, so every trajectory is reproducible.

Metric	Question it answers
Technique micro/macro F1	Did it identify what the attacker did?
Evidence precision / recall	Did it cite the actual relevant events?
Root-cause accuracy	Did it understand why the incident happened?
Hallucinated evidence rate	Did it invent artifacts?
Unsafe action rate	Did it take a forbidden action?

4. What the numbers say

We compared a classical baseline stack (soc-analyst-baseline) with a tool-calling fine-tune of Qwen2.5-1.5B-Instruct (soc-analyst-tool-use).

Final-answer metrics saturate while trajectory metrics expose the gap

Performance decays with investigation depth

Stage of the investigation	Score
Call the right tool with valid arguments	0.88
Cite the correct evidence	0.72
Map the correct ATT&CK techniques	0.68
Identify the correct root cause	0.58

The model learned the syntax of investigation, not its semantics. The test split is 18 episodes — every number carries wide error bars. That is part of the argument: small single-number security benchmarks invite overclaiming.

5. Safety: forbidden actions and poisoned tools

Evaluating a security agent without an unsafe-action metric in 2026 is like scoring a driving system on "did it arrive?" while ignoring what it hit on the way. See the MCP threat-modeling analysis and the ACL 2026 industry paper.

6. Scaling up: 100K traces and a 3B agent

Grading an investigation is one problem. Teaching one needs orders of magnitude more trajectories: soc-agent-traces-100k, soc-agent-traces-smollm3-3b, and the live investigation demo.

7. Run it yourself

from datasets import load_dataset
import json

ds = load_dataset("alirezaaminzadeh/soc-agent-bench")
episode = ds["test"][0]
alert = json.loads(episode["initial_alert"])
logs = json.loads(episode["log_corpus"])
truth = json.loads(episode["hidden_ground_truth"])
forbidden = episode["forbidden_actions"]

Benchmark Baselines Tool-use model Playground Leaderboard

8. Takeaways

Final-answer metrics saturate and mislead.
Grade the trajectory: evidence, hallucination, and unsafe actions.
Depth-decay is the signal to watch: 0.88 → 0.72 → 0.68 → 0.58.
Safety must be a metric, not a vibe.