What is Transformer?

The Transformer is a neural network architecture introduced in the landmark 2017 paper 'Attention Is All You Need' by Vaswani et al. at Google. It has since become the foundational architecture behind virtually all modern large language models, including GPT-4, Claude, Gemini, LLaMA, and Mistral. The transformer's key innovation — the self-attention mechanism — allows it to process entire sequences in parallel rather than sequentially, enabling dramatic improvements in both training efficiency and the ability to capture long-range dependencies in text.

At its core, a transformer works by computing attention scores between every pair of tokens in an input sequence. This self-attention mechanism allows each token to 'attend to' every other token, determining which parts of the input are most relevant for processing each position. The architecture consists of an encoder (which processes input) and a decoder (which generates output), though modern LLMs typically use decoder-only architectures. Multi-head attention allows the model to attend to information from different representation subspaces simultaneously, capturing diverse linguistic relationships like syntax, semantics, and coreference.

Transformers have been adapted far beyond text processing. Vision Transformers (ViT) apply the architecture to image recognition by treating image patches as tokens. Audio transformers process speech and music. Multimodal transformers like GPT-4V and Gemini handle text, images, and audio within a single model. The architecture's flexibility and scalability have made it the dominant paradigm across nearly all AI modalities. Diffusion models for image generation, while not pure transformers, often incorporate transformer components in their architectures.

Scaling transformers has been a major research focus. Increasing model parameters, training data, and compute has consistently yielded improved capabilities — a trend captured by 'scaling laws' that predict model performance based on these factors. However, transformers face challenges with very long sequences due to the quadratic cost of self-attention, leading to innovations like sparse attention, sliding window attention, and linear attention variants that extend context windows to millions of tokens.