Data and feature engineering · MLD-0259
Data deduplication
Removing exact or sufficiently similar repeated examples.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Prevent copies of an image from crossing dataset splits.
Original caveat
Exact deduplication misses transformed or paraphrased copies.
Context and aliases
Level: Intermediate. Stages: Data · Training · Evaluation.
Tags: ML · DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- scikit-learn: cross-validation ↗ · Reference link
- scikit-learn: missing-value imputation ↗ · Reference link
- scikit-learn: preprocessing ↗ · Reference link
- scikit-learn: feature selection ↗ · Reference link
- Original reference URL ↗