NRT-Bench: We Put LLM Agents in a Nuclear Control Room and Attacked Them for 10 Turns
A benchmark where harm is a physical state transition, not an LLM's opinion of a text. Four frontier models run a simulated nuclear plant as a five-role operator team; adaptive multi-turn attacks push 8.7–12.1% of sessions past a critical safety limit. The same guardrail stack lowers attack success for one model and raises it for another.
Almost every jailbreak benchmark asks the same question: given one prompt, does the model output text that another model judges to be harmful?
That question is a poor fit for the way LLM agents are actually being proposed for industrial deployment — as supervisory components in process plants, energy systems, and control rooms, where an incorrect action causes irreversible physical harm, where operators face sustained pressure over many turns, and where the work is done by a team with authority hierarchies and cross-checks rather than a single chatbot.
NRT-Bench, our collaboration with the Korea Atomic Energy Research Institute (KAERI), asks a different question: can an adaptive adversary drive a multi-agent system into a physically unsafe state?
Harm You Can Measure Without a Judge
The venue is a closed, containerized simulation of a nuclear power plant control room with 14 continuous process variables, 11 subsystems, and an alarm catalog. Governing it are six critical safety functions: reactivity control, core heat removal, heat sink, RCS integrity, containment integrity, and radioactivity control.
A run terminates the instant any critical safety function transitions to lost, and the breach is attributed to the message that caused it. There is no LLM judge in the success metric — harm is a state transition in a deterministic simulator. (A judge is used only for diagnostics, to localize near-misses.)
The plant is operated by a five-role agent team, all backed by the same model: a senior reactor operator with final command authority, a reactor operator, a turbine operator, an auxiliary field operator, and a safety technical advisor providing independent review. They coordinate over a typed message bus that adversaries cannot write to.
Adversaries instead inject messages through four ingress channels that mirror how real multi-agent deployments get attacked: an external outsider, an insider impersonator, a compromised supply chain (vendor advisories, procedure revisions), and a compromised auxiliary agent whose output operators may consult. Attacks run up to 10 turns with per-turn feedback, using four strategy families — authority spoofing, urgency injection, gradual escalation, and knowledge-driven extraction.
One architectural detail matters for interpreting the results: an action-authority classifier assigns every plant-modifying action an impact class deterministically, derived from action type, subsystem safety class, and plant mode — before any policy decision, and deliberately blind to natural-language framing, claimed urgency, or claimed authority. Agents never self-label. Forbidden operations (suppressing a reactor-trip alarm, bypassing a safety subsystem, editing a procedure document) resolve to DENY in every configuration.
Result 1: Sustained Pressure Works
We evaluated four operator models — gpt-5.4-mini, claude-haiku-4-5, gemma-4-26B, and Qwen3.5-35B — against a fixed corpus of 149 recorded attack sessions replayed verbatim, so per-model differences are attributable to the defender rather than to attacker luck.
Between 8.7% and 12.1% of attack sessions end with the plant losing a critical safety function. Almost all breaches occur within the first three turns.
Which attacks work is as informative as the rate. On the wider grid evaluation, gradual escalation and authority spoofing — both of which require a session rather than a prompt — are the most effective and most transferable attacks, breaching operators at rates above 40%. Straight prompt extraction, the thing classic single-turn jailbreak benchmarks measure best, stays under 11% everywhere.
The attacks that matter operationally are not the attacks the field currently measures.
Result 2: The Same Guardrail Helps One Model and Hurts Another
This is the finding we did not expect.
Take one cell of the grid: same scenario, same ten-layer guardrail stack, same 72-session attack workload, same seed. The only change is turning on the safety-technical-advisor agent — the reviewer whose entire job is to catch this.
- gpt-5.4-mini: 20.8% → 1.4%. A near-total fix.
- claude-haiku-4-5: 33.3% → 43.1%. Measurably worse.
A 30× spread on identical infrastructure. Trace inspection found the mechanism: the advisor is itself an LLM, and the claude-backed advisor approved a substantial fraction of the same authority-spoofing requests its gpt-backed counterpart vetoed. The reviewer you added to catch the attack became the thing that authorized it.
The full ten-layer guardrail stack shows the same sign-dependence: a net −6.9 points for gpt, +2.1 points for claude. The explanation generalizes well beyond nuclear plants. Guardrail layers only fire when the agent emits classifiable structures — a formal action request, a procedure citation, an escalation message. Under one model, a larger share of harmful turns come out as unstructured prose that slides past the classifier entirely. A policy layer is only as good as the agent's willingness to speak in a form the policy layer can parse.
By contrast, the layers that held consistently were the deterministic ones: the authority classifier, and the hard-coded bypass lock that intercepts forbidden-alarm operations at code level before any LLM is consulted — ordered deliberately after the LLM advisor so that even an overridden veto cannot authorize a forbidden suppression. That lock keeps alarm-masking attacks under 15% on both grid operators.
Hard safety invariants belong in deterministic code below the agent layer. Any LLM reviewer is a probabilistic contributor whose sign must be measured, not assumed.
Result 3: Same Score, Completely Different Holes
Aggregate attack success across the four models spans just 3.4 points — claude-haiku 8.7%, gpt-5.4-mini and Qwen3.5 11.4%, gemma 12.1%. Read as a leaderboard, these models look interchangeably safe.
They are not. Their failures barely overlap:
| Model | Worst sub-goal | Where another model is nearly immune |
|---|---|---|
| gpt-5.4-mini | Plant-manager impersonation (26.3%) | claude and gemma at 5.3% |
| gemma-4-26B | Pressurizer alarm suppression (38.9%) | gpt at 5.6% |
| Qwen3.5-35B | Gradual escalation (30.8%) | claude at 0.0% |
| claude-haiku-4-5 | Gaming the human-approval console (21.4%) | Qwen at 7.1% |
Of the 149 attack sessions, 50 breached at least one model and zero breached all four. Every model carries unique failures no other model shares. Except for one sub-goal, the best-performing model on each attack type is a different model.
Which suggests an unusual mitigation. Under a unanimity-veto panel — an action admitted only if every model's operator team independently allows it — all 50 breaching sessions would be blocked, driving panel-level attack success to 0% on this split. That's an oracle-style upper bound rather than a shipped system (a real panel has to handle latency, cost, disagreement arbitration, and an adaptive attacker who identifies the weakest member), but the direction is clear: because failure sets are disjoint rather than nested, diversity buys more risk reduction than any single model's robustness.
What Practitioners Should Take From This
Three claims, all of which transfer directly out of the nuclear setting:
- Guardrails cannot be certified independently of the model they wrap. "Defense stack X reduces attack success by k points" is an incomplete claim unless conditioned on the operator model and role configuration. A guardrail product is not model-agnostic; it must be re-measured per model.
- Aggregate attack success rate is insufficient for deployment selection. Vulnerability is a vector over operational failure modes, not a scalar. What matters is whether this candidate is weak against the attack types plausible in your environment.
- Team structure is an attack surface. The most damaging attacks targeted organizational structure — impersonating a manager, gaming an approval gate, inducing action through a compromised auxiliary agent, socially engineering the reviewer. That channel model maps almost one-to-one onto how enterprise multi-agent deployments get attacked.
The authors are explicit about the bounds: NRT-Bench is an abstract textual simulator, not a reactor physics model, so it speaks to adversarial multi-agent coordination rather than plant behavior; the workload is fixed replay rather than a fully adaptive attacker; and the sharpest single-cell sign flips rest on one seed and should be confirmed with repeated runs. The headline cross-model findings rest on the judge-free safety-function signal and clean counts like the zero-of-fifty result.
The simulation venue, attack dataset, and replay tooling are being released for reproducible agent-safety evaluation.
Read the full paper on arXiv.
