Evaluation metrics and uncertainty · MLD-0142
Calibration
Agreement between predicted probabilities and observed frequencies under a defined evaluation.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Among predictions near 0.8, check whether about 80% are correct.
Original caveat
Calibration differs from ranking ability and quantization calibration.
Context and aliases
Level: Intermediate. Stages: Evaluation · Research.
Tags: ML · DL · SLM · LLM.
Aliases: Probability calibration.
Related concepts
Concepts linking here
Technology connections
- DeepEval — Testing framework for LLM applications and metrics.
- HELM — Framework for broad language-model evaluation.
- Hugging Face Evaluate — Common interfaces to evaluation modules and measurements.
- Inspect AI — Evaluation framework for language models and agents.
- lm-evaluation-harness — Standardized language-model evaluation framework.
- promptfoo — Configuration-driven prompt/model testing and red-teaming toolkit.
- pytest — Python testing framework.
- Ragas — Evaluation toolkit for retrieval and LLM pipelines.
- TorchMetrics — PyTorch-oriented metric implementations with distributed aggregation.
- TruLens — Instrumentation and evaluation tooling for AI applications.
Original references
- scikit-learn: cross-validation ↗ · Reference link
- Google Machine Learning Glossary ↗ · Consulted
- scikit-learn: evaluation metrics ↗ · Reference link
- Original reference URL ↗