Systems and efficient execution · MLD-0208
Context parallelism
Partitioning long-context computation across devices while preserving needed communication between context partitions.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train with contexts too long for one device.
Original caveat
Definitions overlap with some sequence-parallel approaches.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: CP.
Related concepts
Concepts linking here
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗