Interpretability and mechanistic analysis · MLD-0014
Activation patching
Replacing selected activations with values from another run to test effects on behavior.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Restore a corrupted run using clean hidden states.
Original caveat
Patches can create off-distribution internal states.
Context and aliases
Level: Advanced. Stages: Research · Evaluation.
Tags: ML · DL · SLM · LLM · Interpretability · Safety.
Aliases: Causal tracing (interventions).
Related concepts
Concepts linking here
Technology connections
- Adversarial Robustness Toolbox — Toolkit for adversarial ML evaluation and defenses.
- Captum — PyTorch model-interpretability library.
- Fairlearn — Tools for assessing and mitigating group fairness issues.
- garak — Language-model vulnerability probing toolkit.
- LIME — Local surrogate-model explanation toolkit.
- NeMo Guardrails — Toolkit for programmable controls around conversational applications.
- NNsight — Library for inspecting and intervening in neural-network computation.
- PyRIT — Python framework for AI red-team automation.
- pyvene — Library for interventions on model representations.
- SAELens — Tooling for training and analyzing sparse autoencoders.
- SHAP — Additive feature-attribution methods and tooling.
- TransformerLens — Framework for inspecting and intervening on supported Transformer internals.
Original references
- Captum model interpretability documentation ↗ · Reference link
- Tracing the thoughts of a large language model ↗ · Consulted
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning ↗ · Consulted
- SHAP documentation ↗ · Reference link
- Original reference URL ↗