Tokenization and text representation · MLD-0372
EOS token
A marker indicating a sequence or generation should end under the model's convention.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Stop decoding after an EOS token.
Original caveat
Chat formats may distinguish end-of-turn and end-of-sequence markers.
Context and aliases
Level: Intermediate. Stages: Data · Training · Inference.
Tags: NLP · SLM · LLM.
Aliases: End-of-sequence token.
Related concepts
Concepts linking here
Original references
- Hugging Face: chat templates ↗ · Reference link
- Transformers glossary ↗ · Consulted
- Hugging Face: tokenization algorithms ↗ · Consulted
- Original reference URL ↗