AI Agent Architecture: The Pieces of the System and How They Fit

When an executive asks me to explain the architecture of an AI agent, I start with what it is not: it is not a language model with a good prompt. An agent that executes real work is a system with several pieces, each with a distinct job and a distinct owner for its upkeep. The model reasons and decides, orchestration distributes the work and sequences the steps, tools connect to your systems. Memory and retrieval supply the context, evaluations measure whether it does the job well, observability leaves a trace of what it did, and human controls set how far it can go alone. One of them fails and the whole system stops deserving your trust.

This page is the index for the technical cluster: I present each piece in two paragraphs and link to the article that develops it, so an evaluator grasps the whole before dropping into detail. The order roughly follows the path a case travels from the moment it arrives to the moment it closes.

The model: the piece treated as a utility

The language model is the reasoning engine: it interprets the case, decides the next step, and drafts the result. It is also the easiest piece to replace, and that is exactly the most important design point. The models from the different providers — Anthropic, OpenAI or Google as a category — improve at a pace no client company can match, and shift in price and capability every few months. A sensible architecture treats the model as a utility: selected per task by quality, cost, latency and risk, and replaceable without rebuilding the process.

That forces a double decision. First, which model you assign to each task in the flow, because not all of them need the most expensive one: pulling a figure off an invoice and drafting a delicate reply to a client do not ask for the same thing. I develop that criterion in choosing the right LLM for each task. Second, which options you have available and how they compare for business use, which is the map I draw in LLMs for business.

Orchestration: who does what, and in what order

A real case is rarely resolved in a single step. You have to read an input, query two systems, apply a rule, maybe hand work to another specialized agent, and consolidate the result. Orchestrating is coordinating all of that: distributing the work between agents and tools, sequencing the steps, handling failures and retries, and deciding when a person steps in. Without orchestration you have a model that answers. With it you have a system that works.

Orchestration is a piece large enough to warrant its own article: what it means precisely and how to think about it is in AI agent orchestration. And because there are known ways to structure it depending on the problem — steps in a chain, work in parallel, a supervisor that distributes, a checker that reviews — I have catalogued those forms with their when and their risk in AI agent orchestration patterns.

Tools and integrations: how it touches your systems

An agent that only converses is useless in operations. The value appears when it acts: it queries the ERP, writes to the CRM, reads an email, posts a journal entry. That happens through tools, which are the concrete functions the agent can invoke, each with explicit permissions for what it may read and what it may write. The quality of this layer sets the ceiling of the system, because an agent is only as capable as the actions you have given it, and only as safe as the limits on those actions.

The classic problem is that every integration used to be built bespoke, one by one. Standardizing this layer through a common protocol for connecting agents to tools and data sources cuts that cost and makes integrations portable across models. I explain it without jargon in MCP, the Model Context Protocol.

Memory and knowledge: where the context comes from

The model on its own knows nothing about your company and remembers nothing from two steps ago. The architecture solves this with two distinct mechanisms worth not confusing. Memory and context manage what the agent carries within a task and across tasks: the state of the case, what has been decided so far, what was learned from earlier interactions. That mechanism, and why the context window is not real memory, is what I cover in memory and context in agents.

Your company’s knowledge is another thing: your documents, policies, catalogues and data, which the agent needs to consult without them fitting in the prompt. The technique for that is retrieval: finding the relevant fragment in your document base and handing it to the model at the moment of deciding. Applied to enterprise data, with its problems of permissions and freshness, it is what I cover in enterprise RAG.

Evaluations: how you know it works

This is the piece most people skip and the one that separates a demo from a production system. Evaluating an agent means measuring its quality with test cases that have known answers, thresholds that define “good enough”, and a systematic review of failures. Without evaluations you have no serious way to say the agent works. You have an impression, and impressions do not survive a board.

Evaluations are not a one-off exam at launch: they are run again every time the model, the prompt or the process changes, because any of those changes can degrade something that worked before. How they are built, what gets measured and against which thresholds is the content of AI agent evaluation. It is also the piece that gives the board the evidence to loosen human supervision without betting blind.

Observability and traces: what it did and why

When an agent decides something in production, someone will ask why. Observability is the layer that answers: it records what the agent received, what steps it took, which tools it invoked, which rule it applied, and what result it produced. It is the difference between “the system got it wrong” and “the system applied this rule with this data and produced this result, here is the trace”. Without it there is no debugging and no defensible audit.

This layer is not optional in a serious operation: it is what lets you improve the system with evidence instead of opinion, and what an auditor will demand. I cover it in detail in AI agent observability and traceability. It connects to evaluations — traces feed the test cases — and to governance, which is where this evidence acquires consequences.

Human controls: how far it goes alone

No serious agent operates without limits. The architecture defines at which points a person reviews, approves or takes the decision, and which cases escalate automatically for exceeding the agent’s mandate. Where those control points sit is a business decision, not a technical one. It governs the system’s real risk and moves over time, tightening where the data shows failures and loosening where it shows reliability.

There is a distinction worth being clear about from the design stage: whether the person is inside the loop approving before each sensitive action, or on the loop supervising and able to intervene. Both are legitimate depending on the risk of the case, and I separate them in human in the loop. It is the piece that turns a powerful system into one a board can approve.

Where it runs and how it is built

Two structural decisions cut across the whole architecture. The first is where it runs: in the cloud or inside your perimeter. It affects cost, control, latency and what your regulation demands about where your data lives. The criterion for deciding, without dogma, is in on-premise vs cloud AI agents.

The second is how it is built: with visual automation tools or with agents made bespoke on your rules. Each path has a different ceiling, and confusing them is a frequent source of pilots that never scale. I develop the comparison in no-code automation vs custom agents.

How the pieces fit

The architecture is not the sum of these pieces, but how they connect. The model decides with the context memory gives it and the knowledge retrieval gives it, and it acts through tools with permissions. Orchestration sequences all of that and calls a person when it should, evaluations say whether the whole works and observability records what it did. Removing a piece does not leave the system simpler, it leaves it broken in a way you will not see until a real case reveals it.

The part none of these pieces solves on its own is the one specific to your company: your rules, your exceptions, your permissions, your integrations. That definition is what has to be encoded so the system executes your operation and not a generic one, work I argue for in the operating model as code. And the framework that governs who may do what, and with what evidence, is AI agent governance. If you want the level above — what these systems are and how they are adopted before dropping into the architecture — the entry point is AI agents for business.

Frequently Asked Questions

What components make up an AI agent’s architecture?

A language model that reasons and decides, an orchestration layer that sequences steps and distributes work, tools and integrations to act on your systems, memory and retrieval for context and knowledge, evaluations to measure quality, observability to leave a trace, and human controls to set the limits. Each piece does a distinct job and none is dispensable in production.

Can you build an agent with just a good model and a good prompt?

For a demo, yes; to operate, no. A model with a prompt answers, but it does not act on your systems, does not measure its own quality, leaves no auditable trace, and has no explicit limits. Those pieces are what turn a prototype that impresses in a room into a system a board approves for production.

Which part of the architecture has to be built bespoke?

The infrastructure — model, orchestration, integration protocols, observability — tends to be reusable and increasingly standard. What is specific to your company is the rules, the exceptions, the permissions and the concrete integrations: that layer is encoded case by case because it describes how your operation works, not a generic one.

Where does a technical evaluator advising a board start?

With the three pieces that kill the most pilots and get looked at the least: evaluations, observability and human controls. If a provider cannot show you how it measures quality, how it traces every decision, and where it escalates to a person, the rest of the architecture is beside the point, because you will not be able to defend the system to your board or to an auditor.