Interpretability and mechanistic analysis · MLD-0017

Activation-based monitoring

Using internal model signals to detect specified states, risks, or behaviors.

Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

What an activation monitor observes

An activation-based monitor reads numerical internal states—such as a transformer's residual stream at a chosen layer and token—and produces a score for a risk hypothesis. Unlike an output-only reviewer, it can examine representations before the model finishes speaking or acting. It is a detector trained or selected against labeled examples, not a direct window into hidden intent or guaranteed access to every private computation.

A minimal linear-probe mechanism

Collect vectors h(x) from a fixed model layer for labeled positive and negative prompts. One simple direction is d = mean(h_positive) - mean(h_negative); compute score(x) = dot(h(x), d) and choose a threshold using a separate validation set. More flexible probes may be trained, but capacity adds overfitting risk. Fix model version, layer, token position and preprocessing before evaluating; changing any of them can invalidate calibration.

Controlled study: deliberately planted sleeper agents

Anthropic's 2024 research used residual-stream probes on intentionally backdoored sleeper-agent models and reported strong separation for their constructed defection triggers, including AUROC above 99% in some setups. The team explicitly cautioned that these highly salient features may arise from how the backdoors were planted. This does not establish that similarly simple probes will detect naturally emerging deceptive alignment or an unseen class of harmful behavior.

Worked example: sandboxed tool-use monitor

Imagine a test agent with mock filesystem tools and harmless labeled episodes: authorized file edits, clearly unauthorized edits, and confusing-but-permitted operations. Record activations at a fixed pre-action point, train a probe only on the training split, and audit held-out examples with a human-independent ground-truth label. Compare probe alerts to a behavioral monitor reading tool calls. Use a separate test environment so the labels cannot be inferred from superficial file names or prompt templates.

Evaluation: failures matter more than a single AUROC

Report precision, recall, false positives per thousand benign turns, calibration and detection latency at a threshold that an operator could afford. Split tests by task family, model checkpoint and adversarial reformulation; use a chronological holdout and deliberately probe distribution shift. A high AUROC on synthetic backdoors can coexist with too many false alarms or missed rare events in actual operation. Always audit random unflagged traffic to estimate blind spots.

Operational deployment and independent defense

Treat probe alerts as inputs to a review pipeline, not as verdicts about wrongdoing. Combine them with output/tool-use monitoring, sandbox permissions, rate limits and human escalation; avoid letting the monitored agent modify the monitor, its own logs or its scoring threshold. OpenAI described such activation probes as an initial screening layer in a multi-stage internal monitoring system on October 29, 2026, but this is a vendor account of its practices, not an independently established detection guarantee.

Misconceptions, leakage and adaptation

Activation separation can reflect a spurious cue (for example, a trigger token) rather than the safety property we hoped to measure. A probe trained and tested on near-duplicate templates may look excellent but fail under reformulation; gradients from directly rewarding low alert scores may also incentivize evasion. An internal signal correlating with a bad outcome cannot by itself tell whether the model had deceptive intent, or replace behavioral and system-level controls.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Train a probe to flag suspicious internal patterns.

Original caveat

Monitor reliability must be tested under shift and adversarial adaptation.

Context and aliases

Level: Advanced. Stages: Research · Evaluation.

Tags: ML · DL · SLM · LLM · Interpretability · Safety.

Aliases: Internal monitoring.

Related concepts

Original references

Open original record, export or annotate →