Losses and objectives · MLD-0321
Distillation loss
A loss encouraging a student model to match a teacher's outputs or representations.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Match teacher token probabilities.
Original caveat
Teacher errors and biases can transfer to the student.
Context and aliases
Level: Intermediate. Stages: Theory · Training · Evaluation.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- scikit-learn: evaluation metrics ↗ · Reference link
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗