Sequence architectures and attention · MLD-0775
Multi-head latent attention
An attention design compressing key/value information into latent representations to reduce caching cost.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Use compressed attention states during decoding.
Original caveat
The exact projection and positional scheme are architecture-specific.
Context and aliases
Level: Advanced. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: MLA.
Related concepts
Original references
- Transformers glossary ↗ · Consulted
- DeepSeek-V2: Multi-head Latent Attention ↗ · Reference link
- Original reference URL ↗