Optimization and training dynamics · MLD-0638
Learning rate
A scale controlling how strongly an optimizer changes parameters.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Reduce the learning rate late in training.
Original caveat
Effective update size also depends on optimizer state.
Context and aliases
Level: Foundational. Stages: Training · Theory.
Tags: ML · DL · SLM · LLM.
Aliases: LR · step size.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗