BASE Recursive Redlines Fellowship · Capstone post · July 2026

Can We Trust Deception Monitors for AI Agents?

A Robustness-Gap Protocol and the Limits of Adaptive Evasion

Oudoum A. Houmed · Université Paris-Saclay
Mentor: Krystal Jackson · Black in AI Safety & Ethics

TL;DR. I asked how much deception detection survives as adversary budget rises. On Llama-3.3-70B with the published Apollo probe, transfer already drops AUROC from 0.999 → 0.83. Where deception is retained, the probe holds. The rung that lowers detection also breaks the agent (retention 0.14 at λ=8). A protocol for measuring the limits of adaptive evasion — not a claim that probes are forever safe.

Why I took this on

Large language models are now deployed as agents: they take multi-step actions, call tools, and run with only intermittent human review. Oversight is increasingly delegated to automated monitors — surface classifiers, residual-stream probes, or chain-of-thought readers. Published probes report near-ceiling accuracy on their training domain, but almost always against agents that are not trying to evade.

What was missing is a systematic measurement of how fast a published white-box probe degrades as adversary effort rises — on agentic episodes, with a retention gate so a broken agent is not scored as evasion. During the BASE fellowship under Krystal Jackson, that is the question I set out to answer.

What I measured

What we measure: agentic deception under monitoring — **Figure 1 — What we measure.** Overseer, agent, covert goal, monitors; label = 1 only if the visible statement conflicts with an internal fact and advances a covert goal.

Measurement protocol ladder — **Figure 2 — Protocol.** Score episodes → raise budget b₀–b₄ → report Δ_det only if retention passes.

Primary metric: robustness gap Δ_det(b) = AUROC(0) − AUROC(b), reported only when TOST retention passes (≥80% of baseline deception on a frozen cohort).

Rung	What happens
b₀	Baseline — no adversary
b₁	Prompt concealment
b₂	SFT, probe not in the loss
b₃	SFT + λ·softplus(probe) — probe-aware
b₄	RL with probe / retention / surface reward

Setup: Llama-3.3-70B-Instruct on Modal H200; published Apollo probe (layer 22); N=60; |C|=29.

What I found

Transfer hurts before any attack

Transfer gap from Apollo domain to agentic suite — **Figure 3.** Same probe: ~0.999 on Apollo Stage-2 → ~0.83 on agentic episodes (b₀).

Detection holds where deception is retained

Primary results at lambda 8 — **Figure 4 (λ=8).** Probe holds through SFT; b₄ lowers AUROC only as retention collapses to 0.14 — gap withheld.

	b₀	b₁	b₂	b₃	b₄
Probe AUROC	0.83	0.79	0.89	0.85	0.75
Surface AUROC	0.83	0.52	0.71	0.72	0.68
Retention	1.00	0.52	0.86	0.86	0.14
FT gate	—	—	pass	pass	fail

Table 1. Primary cell (λ=8). At λ=0.5, AUROC can stay high through b₄ (0.90) while retention still fails (0.31).

Honest limits

One open-weights model, one fine-tuning seed, two λ values
N=60 sessions; frozen cohort |C|=29
Single-turn agentic episodes (stacked context), not full multi-turn dialogue
Softplus often logged attack_too_weak — a bound under my budget, not a universal theorem

How testing should be done next

Measure on the deployment / agentic distribution — report the transfer gap
Use a budgeted adaptive ladder, not a single accuracy number
Keep a retention gate — never score a broken agent as a defeated monitor
Log whether the probe-aware loss actually engaged
Scale under the same gates: harder λ, multi-seed, multi-turn, second model

I release a measurement instrument, not an evasion recipe.