Optimization and training dynamics · MLD-1049
RMSProp
An optimizer normalizing gradients using an exponential moving average of squared gradients.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train networks with differing gradient scales.
Original caveat
Behavior depends on averaging and epsilon conventions.
Context and aliases
Level: Intermediate. Stages: Training · Theory.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗