Sequence architectures and attention · MLD-0774
Multi-head attention
Combining several attention heads with separate learned projections.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Mix different contextual information channels.
Original caveat
More heads do not automatically yield better performance.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: MHA.
Related concepts
Concepts linking here
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗