Tokenization and text representation · MLD-1230
TF-IDF
Weighting terms by their document frequency and relative rarity across a corpus.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Build sparse text features for retrieval.
Original caveat
Tokenization and formula conventions affect weights.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: Term frequency-inverse document frequency.
Related concepts
Concepts linking here
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗