Optimization and training dynamics · MLD-1318
Warm restarts
Periodically increasing the learning rate after decay cycles.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Explore multiple optimization basins.
Original caveat
Restarts may or may not reset optimizer state.
Context and aliases
Level: Intermediate. Stages: Training · Theory.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗