Systems and efficient execution · MLD-0571
Inter-token latency
The elapsed time between successive generated output tokens, or an average of that interval.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Measure smoothness of streamed responses.
Original caveat
Averages can hide stalls and long-tail delays.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: ITL · Time per output token · TPOT.
Related concepts
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗