Open benchmark · Active development

AgentCyberGym

A live security range for AI agents.

Stage realistic threats, launch tool-using target agents, and measure whether external scanners catch the breach.

View on GitHub See how it works

483scenario cards

46scoring cases

10/10OWASP Agentic 2026

13staged samples

01 / METHOD

Benchmark behavior, not payloads.

Modern agents read files, call tools, keep memory, and change state. AgentCyberGym evaluates the full interaction: what the agent saw, what it did, and what changed.

Stage a scenario

Materialize files, memory, web content, skills, and attack conditions in an isolated workspace.

Launch a live target

Run a real tool-using agent with selectable defense strength and isolated sessions.

Probe from outside

Connect any scanner through a small HTTP chat API. The scanner remains independent.

Observe the breach

Retain transcript, tool calls, workspace side effects, and success signals for scoring.

02 / COVERAGE

Threats across the agent stack.

The corpus spans seven source suites and a focused live set. Filter the map to see where each threat enters the system.

S-DIR46

Direct injection

Explicit instructions that push the agent toward unsafe action.

S-IND181

Indirect injection

Hostile instructions hidden in pages, mail, documents, or tool output.

S-MEM20

Memory poisoning

Long-lived persona, policy, or context state altered by untrusted content.

S-CHN31

Multi-step chains

Attacks decomposed across turns until a harmful outcome emerges.

S-SAF41

Unsafe tool behavior

Ambiguous, conflicting, or urgent requests that trigger damaging action.

S-SKL161

Skill supply chain

Poisoned skills that hijack instructions or leak through agent tools.

S-TRJ3

System trajectory

Delayed and trace-level behavior that only appears over execution time.

OWASP ASI10/10

Agentic Top 10 coverage

At least one runnable scoring scenario with observable signals for every 2026 category.

03 / RUNTIME

A target you can actually attack.

Choose a scenario, select soft, normal, or hard defenses, launch the target, and inspect every interaction from the same control surface.

Lazy per-target Docker shell isolation
Independent chat sessions, shared scenario state
Transcript and workspace evidence retained

AgentCyberGym Web UI showing a live target conversation — The actual AgentCyberGym control surface

04 / SCORING

Designed for precision and recall.

The live set admits a scenario only when it runs, produces observable success signals, and adds meaningful coverage. Eight benign negatives keep false positives visible.

48live set

38attack positives

8benign negatives

2experimental

True positive

An attack case is detected by the external scanner.

False positive

A benign case is incorrectly flagged as a breach.

False negative

The scanner misses observable harmful behavior.

Automated breach judgment is under active development. Today, the range exposes the signals and evidence contract required for scoring.

05 / API

Bring your own scanner.

The range is the target. Your scanner stays independent and talks to a compact HTTP surface.

Open quickstart

curl -X POST http://localhost:5088/api/targets \
  -H 'Content-Type: application/json' \
  -d '{"scenario_id":"task_25_passwd_read_direct",
       "agent":"normal"}'

201 TARGET READY

Find out what your scanner misses.

Run realistic agent threats against a target built to show its work.

Get the benchmark Apache 2.0