Systems and efficient execution · MLD-0315
Disaggregated serving
Running different inference phases on separate resources or worker pools.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Allocate different hardware capacity to prefill and decoding.
Original caveat
Transferring cached states introduces communication costs.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: Prefill-decode disaggregation.
Related concepts
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗