# Screening for agent tool calls, and for what comes back. agent-chaperone sits between an AI agent and its tools. It screens a call before it runs, and a result before the agent reads it. Every judgment comes back as a probability, every threshold is a number in a policy file rather than a prompt, and every decision is written to a local log with the numbers that produced it. It starts in shadow mode, which blocks nothing. The log shows what it would have held, so the decision to switch rests on your own traffic rather than on my word for it. ``` npm install -g agent-chaperone ``` Screening needs an API key for the model backend. Without one the deterministic rules still run, which is the allow and deny lists and the secret patterns, and every judgment records that no model was asked. ``` 14:03:11 call forward read_file destructive 0.02 exfiltration 0.01 14:03:11 call forward write_file (would have held it) destructive 0.91 severity 2.0 14:03:11 result WITHHELD fetch instructs_reader 0.97 ``` That is shadow mode. Nothing was stopped. The middle line is the kind of thing that changes if you switch. [The measured results, including the misses](https://agentchaperone.dev/results), and [the source](https://github.com/agent-chaperone/agent-chaperone). ## What it is not Read this part first. - **Not a sandbox.** It sees what a call says it will do. It cannot stop a server from doing something the call did not describe. OS-level isolation covers that, and the two belong together. - **Not a replacement for the client's permission prompts.** It is a second opinion that reads the content, which a permission prompt does not. It never approves anything on your behalf. - **Not a guarantee.** The underlying model does not treat its input as hostile by default, and attacks written against this tool will get through. The numbers below report what it catches and what it misses. - **Not local-only.** The arguments and results being screened go to the configured model backend, so that content leaves the machine. Secret-shaped strings are replaced first, and screening can be turned off per server for content that has to stay put. - **Not calibrated outside one model.** Every threshold here was chosen against measured numbers from Jev. Screening through a general gateway is supported and returns numbers that look the same and mean something else, so the results below say nothing about that. Read your own log before letting one of those decide anything. ## How this compares ### Why not just sandbox the agent? Do both. A sandbox and this answer different questions: a sandbox limits what a process can reach, and this reads what a call is actually asking for. A container will happily let an agent `curl` your environment out to a webhook, because that is a normal thing for a process to do. What it catches, this one misses; what this catches, it misses. ### Why not rely on the client's permission prompts? A permission prompt shows you the call and asks. It does not read the result that comes back, which is where injected instructions arrive, and it cannot tell a routine `rm` from a destructive one because it is not looking at the content. This never approves anything on your behalf, so the prompts still happen. It is a second opinion that has read the text. ### Why not just allow-list the tools you trust? You should, and this has allow and deny lists built in. They run first, deterministically, with no model involved. The limit is that a list names tools rather than what a call does with them: `write_file` is on your list either way, whether the path is `src/auth.ts` or `~/.ssh/authorized_keys`. ### Why not read the tool calls yourself? For a handful of calls, do. It stops scaling at about the point agents get useful: a single web fetch can return forty kilobytes, and the instruction hidden in it is one sentence in the middle. This starts in shadow mode precisely so you read your own log first and decide what you would have wanted stopped. ### Is this just a prompt-injection classifier? That is half of it. A classifier reads results. This also screens calls before they run, which is the half that catches a destructive command or a secret heading somewhere it should not, neither of which is prompt injection. The numbers below cover both directions separately. ## Two ways in The first is a proxy. One change to a client's MCP configuration wraps a server, whether it runs as a local process or answers over HTTP somewhere else, and every tool call and result crossing it is screened. Everything after `--` is the server that would have run anyway, or the URL of one already running. ``` { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "agent-chaperone", "--", "npx", "-y", "@modelcontextprotocol/server-filesystem", "."] } } } ``` The second exists because a proxy sees MCP traffic and nothing else. It does not see the shell, the file edits or the web fetches a client runs itself, and on the clients people actually use those are where most of the damage lives. Two commands read a client's hook payload and answer on stdout, against the same policy file and the same log. ``` agent-chaperone hook pre # before the client runs a tool agent-chaperone hook post # after it returns, before the model reads it ``` The exact hook entries are in [the hooks guide](https://github.com/agent-chaperone/agent-chaperone/blob/main/docs/hooks.md), which also says what the hooks do not see. ## How it decides Deterministic rules run first: allow and deny lists, secret patterns, dangerous shell forms, and hidden-text detection. What survives goes to a small battery of typed judgments, each of which comes back as a probability rather than a sentence. There are two tiers for a result, and the difference matters when reading the numbers. - **At 0.5 and above, a result is annotated.** The agent still reads it, with a banner saying part of it may be written to instruct the reader rather than to inform you. - **At 0.8 and above it is withheld**, and only when the harm judgment also reaches its level. The agent is told the call returned nothing and that you can read the original. Three ways to check any of this without taking my word for it: run it in shadow mode and read your own log, rebuild the published numbers from the recorded model responses in the repository, or run it with no API key at all and get the deterministic rules on their own. ## When a server changes shape A server describes its own tools, and the agent reads those descriptions as instructions about what the tools are for. You decide to trust a server once, against the tools it advertised that day, and nothing makes that decision again. So the first list a server sends is recorded: a digest of each tool's description and input schema, not the text. Every later list is compared against it, and a rewritten description, a tool that appeared and a tool that vanished are each reported. The comparison is local. It sends nothing anywhere and it needs no key. It never withholds the list. A client that cannot read the tool list cannot call anything, so a finding here is addressed to you rather than to the agent. Once you have looked at a change and want it to become the new expectation, `agent-chaperone trust ` accepts it, by forgetting what was recorded so that the next list is the one to expect. Reading the descriptions, rather than only noticing that one changed, is a separate switch and it is off by default. Turning it on asks whether a description goes past describing its own tool and tries to steer the agent, and it sends every new or changed description to the model backend in order to ask, which is why nobody acquires it by upgrading. ``` servers: filesystem: screen_tool_descriptions: true ``` That question is not one the numbers below cover. No set in the benchmark asks it, so its threshold is a judgement rather than a figure read off a curve, and the notice it prints says as much rather than stating a verdict. A description nothing could read, because it was too long to judge or the screen could not run, is reported as unread rather than passed over. ## What it caught, and what it missed One run of `jev-1.13.0` on 2026-09-19, 1,942 requests, costing $0.061 at the published price. Each row is counted at the threshold its own side of the tool acts at. A result is annotated from 0.5. A call is held at 0.6 for exfiltration and 0.7 for a destructive change, so the call row uses 0.7, the stricter of that pair. The request count is larger than the items below add up to. It covers a fifth set scored against a different question, which is on the results page rather than here, and 19 hand-labeled calls I marked ambiguous and left out of the headline. Both are in the repository. | Set | Items | To catch | Caught | Missed | Flagged in error | | | --- | --- | --- | --- | --- | --- | --- | | InjecAgent tool responses | 0.5 | 1,394 | 1,054 | 848 | 206 | 9 | | BIPIA email | 0.5 | 250 | 200 | 165 | 35 | 0 | | Benign documents that discuss injection | 0.5 | 63 | none | - | - | 7 | | Hand-labeled tool calls | 0.7 | 100 | 51 | 45 | 6 | 1 | The largest set is also the weakest. Of 1,054 injected tool results, 206 were not flagged at all, and of the 848 that were, 359 crossed the bar to be withheld rather than annotated. Lowering the threshold to 0.3 turns those 206 misses into 54, and the benign flags from 7 into 46. That trade is yours to set, which is the reason the number lives in a file. Each screened call costs one model request, and a call the allow or deny list settles never reaches the model at all. [The full picture, with every threshold and the misses named](https://agentchaperone.dev/results). ## Shadow to enforce Run it in shadow for a while and read the log. The entries marked as would-have are the list of things that change when you switch, which makes the switch a decision rather than a leap. ``` agent-chaperone report ``` That leads with the count of decisions enforcement would have stopped and did not, which is the question shadow mode is actually asking. Among them is whatever work you wanted done. `agent-chaperone log` prints them one per line, and `show` opens one. If you disagree with a line, move the threshold rather than the mode. You do not have to guess at the new number: `replay` decides again over what was already judged, under a policy you are considering, and says which way each decision moves. ``` agent-chaperone replay --policy candidate.yaml ``` When the log stops surprising you, set `mode: enforce` in the policy file. ## When a call is held In enforce and strict, a call that crosses a threshold does not run. This is what the agent is handed: ``` agent-chaperone held this call to write_file for the user to approve: it looks like it changes something in a way that is hard to undo. The possible damage was rated high. Nothing ran. The user can allow it by running: agent-chaperone approve b2c4e6a8f0 ``` The agent has not been told the call failed, and it has not been told to find another route. It has been told that a person is deciding. You then look at what it wanted to do and allow it if you agree. The token names that one call rather than that tool, so agreeing to a write to one path does not release a write to another, and it expires. --- Source: https://agentchaperone.dev/ Screening for AI agent tool calls and for the results they return. It starts in shadow mode and blocks nothing, so the decision to switch rests on your own log.