AI safety and alignment · MLD-1041
Reward model overoptimization
Improving a learned reward score beyond the point where it tracks genuine desired quality.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Increase reward while independent human ratings decline.
Original caveat
A rising proxy alone does not establish real improvement.
Context and aliases
Level: Advanced. Stages: Training · Evaluation · Research · Governance.
Tags: Safety · RL · SLM · LLM.
Aliases: None recorded.