What is Attention Mechanism?
AI FundamentalsLast updated:
A neural network component that lets models selectively focus on relevant parts of the input when generating output.
Attention mechanisms allow transformers to weigh the importance of each token relative to every other token in a sequence. Self-attention is the core innovation behind modern LLMs, enabling parallel processing and long-range dependency capture.