Distillation and model compression · MLD-0504
Hard targets
Discrete target labels or selected outputs rather than full soft distributions.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train on the teacher's chosen answer text.
Original caveat
Hard targets discard some teacher uncertainty information.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Deployment.
Tags: DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Distilling the Knowledge in a Neural Network ↗ · Consulted
- PEFT: LoRA conceptual guide ↗ · Consulted
- PEFT: quantization guide ↗ · Consulted
- Original reference URL ↗