Systems and efficient execution · MLD-0217
Continuous batching
Dynamically adding and removing generation requests between inference steps rather than waiting for an entire batch to finish.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Keep a language-model server busy as requests complete.
Original caveat
Throughput gains can interact with per-request latency.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: Iteration-level batching.
Related concepts
Concepts linking here
Technology connections
- BentoML — Model packaging and inference-service framework.
- KServe — Kubernetes-native model inference platform.
- NVIDIA Triton Inference Server — Multi-framework inference serving platform.
- ONNX Runtime — Cross-platform execution engine for ONNX and supported model formats.
- Ray Serve — Scalable model-serving library on Ray.
- SGLang — Serving runtime for language and multimodal models.
- TensorRT — NVIDIA optimizer and inference runtime for neural networks.
- TensorRT-LLM — NVIDIA optimization and execution stack for LLM inference.
- Text Generation Inference (TGI) — Hugging Face text-generation server now archived/maintenance-oriented.
- vLLM — Inference engine and API server for supported language and multimodal models.
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗