Systems and efficient execution · MLD-0184
Collective communication
Coordinated data exchange involving a group of parallel workers.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Synchronize model states during distributed training.
Original caveat
Communication is not free computation.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: None recorded.
Related concepts
Concepts linking here
Technology connections
- Apache TVM — ML compilation and optimization stack.
- CUDA — NVIDIA GPU programming platform and libraries.
- cuDNN — NVIDIA library of neural-network computation primitives.
- CUTLASS — CUDA templates and abstractions for high-performance linear algebra.
- FlashAttention — Memory-aware exact attention kernel implementations.
- FlashInfer — GPU kernels for LLM inference operations.
- IREE — MLIR-based compiler and runtime for ML deployment.
- NCCL — NVIDIA collective-communication library.
- OpenXLA — Compiler infrastructure for ML computation across supported hardware.
- ROCm — AMD GPU computing software stack.
- torch.compile — PyTorch graph capture and compiler entry point.
- Triton GPU language — Python-embedded language and compiler for GPU kernels.
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗