Systems and efficient execution · MLD-0935
Prefix caching
Reusing computation or cached states for an identical shared prompt prefix.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Avoid reprocessing a common system prompt.
Original caveat
Cache validity depends on exact inputs, model state, and implementation.
Context and aliases
Level: Advanced. Stages: Training · Inference · Deployment.
Tags: DL · LLM · SLM · Systems.
Aliases: Prompt caching.
Related concepts
Technology connections
- Amazon Bedrock — AWS service for accessing supported foundation models and AI capabilities.
- Anthropic Claude API — Hosted access to Claude models and supported tools.
- Cohere API — Hosted language, embedding and reranking interfaces.
- Fireworks AI — Managed inference and model-customization platform.
- Gemini API — Google's developer interface for supported Gemini models.
- Gemini Enterprise Agent Platform — Google Cloud managed AI platform; current documentation destination for former Vertex AI material.
- LiteLLM — Provider-normalizing Python library and proxy gateway.
- Microsoft Foundry — Microsoft platform for building and operating AI applications and model access.
- Mistral API — Hosted access to Mistral models and supported document/model services.
- OpenAI Responses API — Hosted interface for model responses and supported built-in tools.
- OpenRouter — Unified gateway to multiple model providers.
- Together AI — Hosted inference and customization platform for supported models.
Original references
- PyTorch Fully Sharded Data Parallel ↗ · Consulted
- PyTorch documentation ↗ · Reference link
- vLLM documentation ↗ · Consulted
- Original reference URL ↗