Sequence architectures and attention · MLD-0433
FlashAttention
An IO-aware implementation of exact attention reducing memory traffic through tiling and recomputation.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train attention models with lower memory overhead.
Original caveat
It optimizes execution rather than changing the attention objective.
Context and aliases
Level: Intermediate. Stages: Training · Inference · Theory.
Tags: DL · SLM · LLM · NLP.
Aliases: None recorded.
Related concepts
Technology connections
- FlashAttention — Memory-aware exact attention kernel implementations.
Original references
- FlashAttention: Fast and Memory-Efficient Exact Attention ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗