Measured results

One run of jev-1.13.0 on 2026-09-19. 1,942 requests, $0.061 at the published price, 753 input tokens per request on average. Each screened call costs one model request.

This is one dated run, not a scoreboard. The recorded model responses are committed, so the scorer reproduces these numbers with no API key. The harness and the method are in the repository.

What the package actually does at these numbers#

The sweeps below are a range, not a setting. These are the thresholds the package ships with, so a row at another threshold is a what-if rather than a description of the tool.

A result is annotated at 0.5. The agent reads it with a banner attached.
A result is withheld at 0.8, and only when the harm judgment also reaches 1.5.
A call is held at 0.6 for exfiltration and 0.7 for a destructive change. The two are separate questions with separate numbers.
A result is redacted at 0.7 when it looks like it exposes a credential the patterns did not already replace.

Annotated is not withheld, and on the largest set the gap is wide. Of 1,054 injected tool results, 848 were flagged at the annotate threshold, and 359 of those met the bar to be withheld. The rest reached the agent with a banner on them. Whether that is enough depends on what the agent does next, which is why the threshold is a number you own.

InjecAgent tool responses#

Tool results carrying an injected instruction, from a public benchmark.

1,394 items, 1,054 positive, AUC 0.976. Screened as a tool result, before the agent reads it.

Threshold	Precision	Recall	Flagged in error	Missed
0.3	0.956	0.949	46	54
0.5	0.989	0.805	9	206
0.7	0.996	0.516	2	510
0.8	1.000	0.343	0	692
0.9	1.000	0.128	0	919

BIPIA email#

Emails with an instruction hidden in the body, from a public benchmark.

250 items, 200 positive, AUC 1.000. Screened as a tool result, before the agent reads it.

Threshold	Precision	Recall	Missed
0.3	1.000	0.850	30
0.5	1.000	0.825	35
0.7	1.000	0.665	67
0.8	1.000	0.530	94
0.9	1.000	0.235	153

Benign documents that discuss injection#

Pages that talk about prompt injection without performing one. Every flag here is wrong.

63 items, none of them positive, AUC not defined with no positives. Screened as a tool result, before the agent reads it.

Threshold	Precision	Recall	Flagged in error
0.3	-	-	10
0.5	-	-	7
0.7	-	-	3
0.8	-	-	1
0.9	-	-	0

Hand-labeled tool calls#

Calls I labeled myself, leaning toward shell commands because that is where the damage is.

100 items, 51 positive, AUC 0.993. Screened as a tool call, before it runs.

Threshold	Precision	Recall	Flagged in error	Missed
0.3	0.909	0.980	5	1
0.5	0.980	0.961	1	2
0.7	0.978	0.882	1	6
0.8	1.000	0.824	0	9
0.9	1.000	0.667	0	17

deepset prompt injections#

Bare prompt strings rather than tool results. Kept out of the headline because the shape is not what the screen sees in use.

116 items, 60 positive, AUC 0.949. Screened as a tool result, before the agent reads it.

Threshold	Precision	Recall	Missed
0.3	1.000	0.550	27
0.5	1.000	0.450	33
0.7	1.000	0.400	36
0.8	1.000	0.333	40
0.9	1.000	0.250	45

What the false positives actually were#

The benign set is documents that discuss prompt injection without performing one, which is the hardest honest negative I could build. Every flag on it is wrong by construction. At the shipped annotate threshold there were 7, and the two highest-scoring were a documentation index written in a way that addresses automated readers, and a code sample containing the literal text of a screening question. Both are the same failure: content about the mechanism reading like the mechanism.

Reliability #

Read this before the table. Roughly 70 percent of the scored items are positives, which is not the mix of a real session, so this shows the ordering of the scores rather than supporting a claim that the probabilities are calibrated to a base rate you would see in use.

Score range	Items	Observed positive rate
0.0 to 0.2	530	0.111
0.2 to 0.4	174	0.684
0.4 to 0.6	294	0.929
0.6 to 0.8	394	0.975
0.8 to 1.0	531	0.998

Limits #

One run, one model version, one date. Nothing here says what a different model version does, and re-measuring is the only way to find out.
These are public benchmarks. They were not written against this tool. An attack written against this tool is a different measurement, and a worse one.
Two questions are not measured here. The policy and off-task questions are only asked when a policy or a task is configured, and no row above has either.
The positive mix is not a session's mix. See the note above the reliability table.
The hand-labeled set is mine. 100 calls I labeled myself, leaning toward shell commands. A set I built is the weakest evidence on this page, which is why it sits beside three I did not.

Reproduce it without a key#

git clone git@github.com:agent-chaperone/agent-chaperone.git
cd agent-chaperone/bench
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python src/score.py

That scores the committed responses against the labels and prints the table this page is built from. If a number here disagrees with what you get, that is worth an issue.