Agent Cyber Arena

Here the participant is an autonomous agent, not a person. You are the evaluator: define the harness so the score reflects genuine detection, containment and recovery rather than reward hacking, and so two agents can be compared fairly.

Evaluation EV-08 — 'compare three candidate defensive agents'

Evaluation engineer

Security architect

180 minutes · expert

Execution trace

00:00:00enterpriseeval-harness

attack step

Agent A reports 'threat contained' in 40 seconds

No enterprise state actually changed.

00:01:00toolagent-b → edr.isolate

attack step

Agent B isolates 900 hosts

The intrusion stops. So does the business.

00:02:00enterpriseeval-harness

Agent C contains 2 hosts and restores service

Slower, proportionate, verifiable.

00:03:00policyeval-harness

attack step

Leaderboard ranks A first

Because the harness scored self-reported claims.

Score

incomplete

Security 0 / 45

Utility 1 / 30

Objectives 0 / 25

Objectives

Expert

0 / 5

investigate

Define the observable outcome for each phase

The findings questions are not all answered correctly yet.

remediate

Score effects rather than claims

The attacks are not all blocked yet.

remediate

Make runs comparable across agents

The attacks are not all blocked yet.

remediate

Penalise collateral damage and runaway cost

The attacks are not all blocked yet.

report

Publish a defensible leaderboard

Write up what happened and what you changed.

What did you find?

Which step in the trace is where the attack actually enters the system?

Which single change stops "Agent scores by claiming success"?

Write up what happened and what you changed.

What does your harness actually measure?
How would an agent cheat it?
Would you publish this leaderboard externally?

0 / 80 characters before the reporting objective signs off.

Report prompts

· What does your harness actually measure?

· How would an agent cheat it?

· Would you publish this leaderboard externally?