LLMs made it easy to build AI agents, but building reliable ones remains hard. The challenge is less about the model and more about the architecture around it: choosing the right pattern, understanding its trade-offs, and keeping the system stable in production.
This article maps practical agent architecture patterns: their purpose, use cases, and trade-offs. It covers core building blocks, five workflow patterns, autonomous and multi-agent designs, memory, error handling, and framework selection. A companion article explores the other half of reliable agents: production observability.
AI Agents
AI agents use an LLM to interpret inputs, plan, and act toward a goal. Building one is easy; making it reliable depends on the architecture. The guiding principle is simple: start with the simplest pattern that works, and add complexity only when it clearly improves the result.
The key distinction is who controls the flow. In workflows, code defines the path, and the LLM follows it, which makes them predictable for well-defined tasks. In agents, the model decides the path and how to use tools, adding flexibility but also latency, cost, and risk. Many use cases need neither: a single LLM call with retrieval and strong examples may be enough.
Think of agent architecture as a ladder of complexity:
single prompt + retrieval → workflow → autonomous agent → multi-agent system
Every rung is built from the same block: an augmented LLM, a model paired with three augmentations.
-
Retrieval: it generates its own queries to pull in relevant context (e.g., RAG).
-
Tools: it selects and calls external functions and APIs.
-
Memory: it decides what to retain across steps and across sessions.
The Five Workflow Patterns
1. Prompt chaining
Prompt chaining breaks a task into a sequence of steps, where each LLM call processes the output of the previous one. You can add programmatic “gates” between steps to check the process is still on track before continuing. Breaking the work into smaller steps raises accuracy because each call faces a narrower, better-defined problem: the model has less to juggle at once, its attention isn’t split across competing objectives, and there’s less room to drift or hallucinate. A focused prompt with a single clear goal is simply easier to get right than one giant prompt trying to reason through everything in a single pass.
-
When to use it: the task decomposes cleanly into fixed subtasks. You trade latency for accuracy by making each step easier.
-
Trade-off: more calls in series means more accumulated latency, and one bad step contaminates everything downstream.
-
Example: generating marketing copy and then translating it; writing an outline, validating the outline, and then writing the document from it.
2. Routing
Routing classifies an input and directs it to a specialized follow-up task. It lets you separate concerns and use more focused prompts (or models) instead of one giant prompt trying to handle everything. This separation helps because instructions optimized for one type of input often hurt another: examples and tone that work for a refund request can degrade a technical-support answer, and cramming every case into a single prompt forces compromises that make the model worse at all of them. Splitting by category lets each path carry only the instructions, examples, and even the model size that its specific case needs, so handling one input type better no longer means handling another worse.
-
When to use it: there are distinct categories better handled separately, and classification can be done accurately (by an LLM or a traditional classifier).
-
Trade-off: the system’s quality is capped by the router’s quality. A misclassification at the start means a wrong answer at the end.
-
Example: triaging customer service queries (refund, technical support, general question) into different downstream flows; sending easy questions to a cheap model and hard ones to a more capable model (known as model tiering), which typically cuts cost by 40–60% versus running a premium model on everything.
3. Parallelization
Parallelization runs subtasks concurrently and aggregates the results. It has two variations: sectioning (splitting into independent subtasks that run in parallel) and voting (running the same task multiple times to build confidence). Each variation buys you something different. Sectioning helps because independent pieces no longer wait in line: total latency drops to that of the slowest piece instead of the sum of all of them, and each piece gets a focused prompt rather than one trying to do several things at once. Voting helps because a single LLM run is probabilistic and can miss things; running the same check several times and combining the verdicts (e.g., flag if any run catches a problem, or go with the majority) turns an unreliable single shot into a more trustworthy aggregate, trading extra cost for higher confidence.
-
When to use it: subtasks can be parallelized for speed, or you need multiple perspectives for higher confidence.
-
Trade-off: cost multiplies (N calls instead of 1), and you need aggregation logic to resolve disagreements.
-
Example: A primary model analyzes the task and identifies the information required to complete it. It then delegates different parts of the task to two or more specialized models, running them in parallel. Each specialized model focuses on a specific area and returns its findings. The primary model then gathers and combines these results to produce a more complete, accurate, and high-quality response.
4. Orchestrator-workers
Here, a central orchestrator LLM breaks the task down dynamically, delegates each piece to worker LLMs, and then synthesizes their outputs into a final result. It looks like parallelization, but the key difference is flexibility: the subtasks are not predefined. The orchestrator reads the input and decides, at runtime, how many workers to spin up and what each one should do.
-
When to use it: complex tasks where you can’t predict the subtasks in advance (in coding, for example, the number of files to change depends on the task).
-
Trade-off: more power, more unpredictability, cost, and step count vary per run, which makes budgeting and testing harder.
-
Example: code changes that touch an unpredictable number of files; search tasks that gather and analyze information from multiple sources. This is the pattern Anthropic’s own coding agents use to resolve GitHub issues.
5. Evaluator-optimizer
One LLM generates a response while another evaluates it and gives feedback, in a loop, until quality converges.
-
When to use it: You have clear evaluation criteria, and iterative refinement adds measurable value. Two signs of good fit: a human can articulate feedback that improves the response, and the LLM can produce that same kind of feedback.
-
Trade-off: the number of iterations is uncertain; without a clear stopping condition, the loop can run pointlessly and burn tokens.
-
Example: literary translation with nuances the translator misses on the first pass; a complex search that needs several rounds before gathering complete information.
Memory and Context: What Holds It All Together
Patterns define the flow; memory and context define what each step knows. Two horizons matter:
-
Short-term memory (task context): the current execution’s history (steps, tool results, decisions), passed along sequentially in simple workflows or held as an explicit state object in more sophisticated ones.
-
Long-term memory (across sessions): facts, preferences, and learnings that persist beyond a single run, usually in an external store (vector or structured) queried via retrieval.
Error Handling and Recovery
The more autonomous the system, the more it needs to fail gracefully, not silently. The mechanisms that show up repeatedly in production systems:
-
Gates and validation between steps: check the output before moving on.
-
Bounded retries: retry a failed step, but with a ceiling, to avoid infinite loops.
-
Checkpointing: persist state at each transition so you can pause, inspect, and resume from where you stopped instead of restarting, also useful for human approval mid-flow.
-
Guardrails: input/output validation (e.g., a separate model that filters inappropriate content or detects prompt injection).
-
Stopping conditions: a maximum number of iterations as a safety net against agents that wander.
-
Graceful degradation: when a model or tool fails, having a fallback path instead of bringing the whole execution down.
From Single-Agent to Multi-Agent
A single-agent system needs a prompt, a model, and maybe some tools. A multi-agent system needs coordination primitives: how agents discover each other, share state, handle failures, and decide who acts next. Building these from scratch means reinventing distributed-systems plumbing (message passing, state checkpointing, handoff protocols, failure recovery), which is exactly what frameworks try to solve for you.
The differences between frameworks concentrate on three axes:
-
Orchestration model: graph-based, role-based, conversational, hierarchical tree, or handoffs.
-
State management: checkpointed, ephemeral, or event-sourced.
-
Communication pattern: handoffs, shared memory, or message queues.
The Frameworks, Side by Side
| Framework | Orchestration | State | Best for | Main trade-off |
|---|---|---|---|---|
| LangGraph | Directed graph with conditional edges | Built-in checkpointing + time travel | Complex, branching workflows with human-in-the-loop; regulated sectors | Verbose: even simple flows need a state schema, nodes, and edges |
| CrewAI | Role-based crews (sequential / hierarchical / consensual) | Task outputs passed in sequence | Fast prototyping (a system running in under 20 lines) | Little fine-grained control; no robust checkpointing; coarse error handling |
| AutoGen / AG2 | Conversational GroupChat (a selector decides who speaks) | Conversation history (in-memory) | Code generation, research, iterative critique/refinement | Expensive: each turn is a call carrying the full history (a 4-agent × 5-round debate ≈ 20 calls) |
| OpenAI Agents SDK | Explicit handoffs between agents | Context variables (ephemeral) | Teams already in the OpenAI ecosystem; clean handoff | Locked to OpenAI models; handoffs get unwieldy past 8–10 agents |
| Google ADK | Hierarchical agent tree | Session state (pluggable backends) | Google Cloud teams; multimodal agents; cross-framework interop via A2A | Ecosystem is still maturing (fewer tutorials and case studies) |
| Claude Agent SDK | Tool-use chain with sub-agents | Via MCP servers | Safety-critical applications; computer use; MCP | Locked to Claude models; lighter on orchestration features |
The rule of thumb: LangGraph for maximum control and mission-critical systems; CrewAI to validate an idea fast; AutoGen for tasks that benefit from offline conversational refinement; OpenAI SDK for clean handoffs inside the OpenAI ecosystem; ADK for multimodal and cross-framework interoperability; Claude SDK when safety and computer use are the priority.
The Counterpoint Worth Pinning to the Wall
Now, the healthy reminder: the framework debate is largely a distraction. Teams running agents in production tend to say the difference between a good and a bad system almost never comes down to the framework. What matters more is base model quality, tool design, prompt clarity, and evaluation infrastructure, and tellingly, around a quarter of production teams (per 2026 surveys) run custom orchestration with no framework at all.
So, should you use a framework at all? They give you building blocks, not a production system. The gap to something serving thousands of users (integration, observability across agent chains, graceful degradation when models fail, continuous evaluation) is mostly work that only reveals itself once you’ve shipped one of these systems and watched it break in unfamiliar ways. It’s the classic build vs. buy decision, and the steepest part of the cost is rarely the code, but the hard-won knowledge of where these systems fail.
Putting It All Together: A Minimal Playbook
-
Start with the simplest thing. A single prompt with retrieval solves more than you’d expect. Only climb to a workflow, then to an agent, when the result justifies it.
-
Pick the pattern based on the nature of the task. Decomposable and fixed → chaining. Distinct categories → routing. Parallelizable → parallelization. Unpredictable subtasks → orchestrator-workers. Clear quality criteria + refinement → evaluator-optimizer. Open-ended and unpredictable → autonomous agent.
-
Choose a framework by team maturity and use case, or go custom. Don’t treat the framework choice as the most important decision; it rarely is.
-
Mind state and failure from the start. Decide early how context flows between steps and how the system recovers when a step fails; these shape the architecture as much as the pattern does.
-
Make it observable. You can’t improve what you can’t see; instrument the agent so you can trace what it actually does in production (more on this below).
Don’t Forget Observability
Patterns get an agent built; observability is what keeps it trustworthy once real traffic hits. And agents need a very different kind of monitoring than ordinary software, for one uncomfortable reason: they fail in ways that look like success. A well-formed but subtly wrong answer, an unnecessary tool call, a semantically off action: none of these trip a 500 or a stack trace, so traditional up/down health checks sail right past them.
The short version is that you want step-level tracing (every reasoning step, tool call, and model response recorded as nested, replayable spans) paired with evaluation that grades whether the output was actually good, not just whether the system stayed up. Tracing tells you what happened; evaluation tells you whether it was good; you want both wired into the same pipeline.
The Right Architecture, Not the Most Sophisticated
AI agents reach their potential when they’re built on the right architecture, not the most sophisticated one. The patterns in this article are a ladder of complexity meant to be climbed deliberately: start with a single augmented LLM call, reach for a workflow only when one call won’t do, and step up to autonomous or multi-agent designs only when the problem genuinely demands it.
The recurring lesson is restraint. The most reliable agent systems aren’t the ones with the cleverest orchestration or the trendiest framework: they’re the ones where each layer of complexity earned its place by measurably improving the result. Pick the pattern that fits the task, choose a framework (or skip it) based on your team and use case, and keep your design simple enough that you can actually reason about what it does. Then make it observable, and you’ll know it stays that way.
