Sequence architectures and attention · MLD-1080
Scaled dot-product attention
Attention using scaled query-key dot products, usually followed by softmax and a weighted value sum.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Compute one attention head.
Original caveat
Scaling and masking are essential parts of the operation.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: SDPA.
Related concepts
Concepts linking here
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗