Distillation and model compression · MLD-0675
Logit distillation
Transferring knowledge by matching teacher output logits or derived distributions.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Minimize divergence between teacher and student predictions.
Original caveat
Matching raw logits and matching probabilities are different objectives.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Deployment.
Tags: DL · SLM · LLM.
Aliases: Response-based distillation.
Related concepts
Original references
- Distilling the Knowledge in a Neural Network ↗ · Consulted
- PEFT: LoRA conceptual guide ↗ · Consulted
- PEFT: quantization guide ↗ · Consulted
- Original reference URL ↗