Distillation and model compression · MLD-0607
Knowledge distillation
Training a student to reproduce useful behavior, probabilities, or representations of a teacher.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Transfer capabilities into a compact model.
Original caveat
The teacher need not be larger, and fidelity is not guaranteed.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Deployment.
Tags: DL · SLM · LLM.
Aliases: KD.
Related concepts
Concepts linking here
Technology connections
- Axolotl — Configuration-driven model fine-tuning toolkit.
- bitsandbytes — Low-bit quantization and memory-saving optimizer primitives.
- Hugging Face PEFT — Parameter-efficient model adaptation methods.
- Hugging Face TRL — Post-training trainers for supervised and preference/RL-style methods.
- LLaMA-Factory — Unified training and fine-tuning toolkit for supported models.
- NVIDIA Model Optimizer — Model optimization tools including supported quantization, pruning and distillation paths.
- OpenRLHF — Distributed RLHF and post-training framework.
- TorchAO — PyTorch architecture optimization and quantization tooling.
- torchtune — PyTorch-native post-training recipes; development has wound down.
- Unsloth — Optimized fine-tuning tooling for supported model families.