LLM Application Architecture
The architecture of an LLM-powered application involves more than the model itself. Understanding the full stack — from input to output — is essential for building reliable systems.
Core components
Input processing. User inputs must be validated, sanitised, and potentially augmented before reaching the model. This includes prompt injection defence, content moderation, and appending retrieved context.
The model call. The actual API request to the LLM, with system prompt, conversation history, and any retrieved documents. This is where token limits, latency, and cost are primarily incurred.
Output processing. Model outputs must be parsed, validated, and potentially post-processed before reaching the user. Structured output validation, content filtering, and citation extraction happen here.
Memory and state. Most LLMs are stateless — each API call is independent. Maintaining conversation context, user preferences, and long-term memory requires explicit state management: either including history in each prompt (expensive at scale) or using a vector database or summary layer.
Orchestration frameworks
Frameworks like LangChain, LlamaIndex, and Anthropic Claude's native tool use simplify building complex LLM pipelines. They handle prompt templating, retrieval integration, tool calling, and output parsing. Evaluate whether the abstraction layer is worth the added complexity for your use case — many simpler applications do not need a framework.
Latency management
LLM calls are slow relative to traditional software operations — typically 1-10 seconds for a full response. Streaming (receiving and displaying tokens as they are generated) significantly improves perceived performance. For batch operations, parallel calls and caching (for repeated identical queries) reduce wall clock time.
LLM applications fail most often at the edges — input validation, output parsing, and error handling — not in the model call itself. Design the surrounding system with the same rigour as the AI components.