Systems and efficient execution · MLD-0448
Fully sharded data parallel
A distributed training approach that shards model state across data-parallel workers and gathers needed pieces for computation.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train larger models with less memory per GPU.
Original caveat
Sharding saves persistent memory but adds communication.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: FSDP.