Sequence architectures and attention · MLD-1095
Self-attention
Attention where queries, keys, and values originate from the same sequence representation.
Expanded guide · editorial review 2026-11-01. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Prerequisites
How it works
Self-attention forms queries, keys and values from the same input sequence. Each query compares with allowed keys, then uses normalized scores to combine values. Several heads can learn different projections. Causal self-attention restricts a position to earlier positions and itself; bidirectional attention can use both sides.
Worked example
Suppose a query has two equal scores and values 2 and 8. Softmax gives weights 0.5 and 0.5, producing 5. Masking the second key makes its weight zero and the output 2. This scalar toy example illustrates weighted averaging, not the full behavior of a trained language model.
The core equation
Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k) + mask) V. Rows of the softmax describe the weights for each query; disallowed positions receive negative infinity before normalization.
What to test
Verify each allowed row sums to one and masked positions receive zero weight. Check tensor shapes and axis selection before optimizing the implementation.
Common pitfall
An attention weight is a routing coefficient. It does not, by itself, establish a causal explanation of an output.
Sources for the expanded explanation
Editorial synthesis checked against the linked primary material; no independent replication is claimed.
Original example
Relate words within a sentence.
Original caveat
The attention mask determines which positions are accessible.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Attention Is All You Need ↗ · Consulted
- Transformers glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗