Transformer Architecture
Large Language Models have become foundational technology for AI-powered products. Understanding the architecture at a conceptual level — without necessarily the mathematics — is essential for anyone building or procuring AI systems.
The transformer architecture
Modern LLMs are built on the transformer architecture, introduced by Google researchers in the 2017 paper "Attention Is All You Need". The key innovation was the attention mechanism: instead of processing text strictly left to right (as earlier models did), transformers can attend to any part of the input simultaneously, weighting the relevance of each token to every other.
This parallel processing enabled transformers to be trained on far more data than previous architectures, and to capture longer-range dependencies in text.
Tokens, not words
LLMs process text as tokens — sub-word units that balance vocabulary size with coverage. "Unforgettable" might be tokenised as ["Un", "forget", "table"]. Most English words are one or two tokens. This matters practically: models have context windows measured in tokens, and token count affects both capability and cost.
Pre-training and fine-tuning
LLM development has two stages. Pre-training involves training on enormous text corpora to predict the next token — this creates a base model with broad knowledge but no particular helpful disposition. Fine-tuning (including RLHF — Reinforcement Learning from Human Feedback) shapes the model into a helpful, safe assistant. The fine-tuned model is what you interact with as a product.
Emergent capabilities
As models scale, they exhibit capabilities that were not explicitly trained for and could not be predicted from smaller models — a phenomenon called emergence. In-context learning (the ability to perform tasks from examples in the prompt alone) appeared as an emergent property at scale. This non-linearity makes frontier model capabilities hard to forecast.
Understanding that pre-training builds knowledge and fine-tuning shapes behaviour is foundational to making good decisions about when to fine-tune a model versus when to use prompting alone.