Systems and efficient execution · MLD-0741
Mixed-precision training
Training with multiple numerical precisions, using lower precision where suitable and higher precision where needed for stability.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Speed training while controlling numerical error.
Original caveat
Using low-precision weights alone does not describe the whole training scheme.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗