Systems and efficient execution · MLD-0169
Chunked prefill
Processing a long prompt in smaller chunks that can be scheduled alongside other inference work.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Reduce interference between long prompts and ongoing generation.
Original caveat
Chunk size affects overhead and scheduling fairness.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: None recorded.
Related concepts
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗