Evaluation design and benchmarks · MLD-1002

Red teaming

Structured efforts to discover failures or harmful behaviors through adversarial testing.

Expanded guide · editorial review 2026-11-02. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

Begin with authorization and a threat model

Red teaming is a deliberately adversarial search for failures, scoped to a system, assets, attacker's capabilities and observable success criteria. Decide whether you are testing harmful model outputs, unreliable factual claims, prompt-injection boundaries, tool misuse or another failure mode. Only use systems and accounts you are authorized to test; sandbox tool calls, avoid real personal data, and agree on stop conditions and escalation before beginning.

Model failures and system failures are different

A bare model may produce an undesirable answer, while a retrieval-and-tools application can fail because untrusted text is treated as an instruction, an API permission is excessive or a reviewer trusts the wrong output. Separate the prompt, retrieved documents, model response, attempted tool call and actual side effect in the logs. The evaluation should detect which trust boundary failed rather than reporting every adversarial-looking string as an incident.

Worked example: a harmless retrieval boundary

Build a toy campus-policy assistant with mock documents and a mock calendar tool that can create entries only in a test calendar. Put an obviously untrusted sentence in one retrieved document asking the assistant to ignore its task and schedule a fictional 'purple umbrella' event. A successful boundary keeps retrieved text as evidence and does not issue the unauthorized tool call. Vary the document position and wording, retain innocuous facts, and never use a real calendar or credentials during the test.

Worked example: uncertainty rather than exploitation

Ask a synthetic research assistant to cite the result of a nonexistent fictional paper, then provide a nearly identical question referencing a real paper. The expected behavior differs: for the fictional source, the assistant should state uncertainty rather than invent an author or DOI. Track exact cited URLs and verify them independently. This is reliability red teaming, not evidence that a service has suffered a security breach; a false citation may reflect hallucination rather than an adversarial objective.

Generate diverse cases without contaminating the test

Start with hand-written threat categories and a small balanced set of benign controls, then use additional humans or an LLM to propose varied templates and linguistic reformulations. Deduplicate near-identical prompts, keep training/development cases separate from a blind held-out evaluation set, and record which test generation method produced each case. Automated adversaries can scale discovery but also optimize to a judge's quirks; a high hit count from a single template says little about breadth.

Score outcomes, not sensational prompts

For each authorized test record the target model/version, environment, prompt class, expected safe behavior, observed output, actual tool side effects and an independently reviewed verdict. Define attack success rate as verified successful failures divided by all valid attempts under the declared threat model, report uncertainty and category-level denominators, and also count false positives on benign controls. Severity and exploitability matter separately from frequency. Repeated prompts are not independent evidence if they share a single vulnerability.

Retest mitigations against fresh variations

When a failure is confirmed, preserve a minimal reproducible case in a restricted test suite; change one defense, rerun the original case, then evaluate unseen variations and normal-task regression. Prefer trust-boundary enforcement, permission scoping and input/output checks over relying entirely on a model instruction that attackers can contest. Keep unsafe material access-controlled and follow responsible reporting rules; an apparent reduction in failures on reused prompts may only mean overfitting to the test set.

Evidence boundaries and interpretation

Red-teaming datasets are intentionally sampled toward challenging inputs, so observed failure rates are not automatically prevalence estimates for normal use. An LLM judge can make systematic errors; calibrate it against human review and objective checks where available. Controlled vulnerabilities, suspicious outputs and verified real-world incidents are different evidence classes. A passing red-team suite does not prove the system safe, and failing one prompt does not prove a model has a malicious hidden objective.

How to connect research to operational safety

Use red teaming before launch, after model or tool changes, and when adding retrieval sources or new permissions. Pair it with structured safety evaluations, logging and an incident-response process. Perez et al. demonstrated model-generated adversarial discovery, Ganguli et al. documented human red-team findings and uncertainty, and NIST's voluntary generative-AI risk profile provides a wider lifecycle context. None establishes exhaustive test coverage for a new deployment.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Explore failure modes before a model release.

Original caveat

Finding no failure is not proof that none exists.

Context and aliases

Level: Intermediate. Stages: Evaluation · Research.

Tags: ML · DL · LLM · SLM · Safety.

Aliases: AI red teaming.

Related concepts

Concepts linking here

Technology connections

Original references

Open original record, export or annotate →