Distillation and model compression · MLD-0417
Feature distillation
Training a student to match selected teacher intermediate representations.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Align hidden features between networks.
Original caveat
Different hidden dimensions may require projection layers.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Deployment.
Tags: DL · SLM · LLM.
Aliases: Representation distillation.
Related concepts
Concepts linking here
Original references
- Distilling the Knowledge in a Neural Network ↗ · Consulted
- PEFT: LoRA conceptual guide ↗ · Consulted
- PEFT: quantization guide ↗ · Consulted
- Original reference URL ↗