How to Deploy an AI Agent

Agents are easy to demo and awkward to deploy, and the reason is a single property: you do not know how long a run will take, because you did not write the path.

Every deployment decision follows from that. Request timeouts, retries, scaling, cost control, state — all the standard web-service answers assume a bounded unit of work. An agent is not one.

The runtime shape you are actually deploying

A conventional request is a few hundred milliseconds of your code. An agent run is minutes of waiting, punctuated by short bursts of your code. The model is thinking, or a tool is calling somebody else's API, and your process is idle.

That has three immediate consequences.

Your request timeout is wrong. A run that legitimately takes four minutes dies against a 60-second gateway timeout, and it dies after spending the tokens. Whatever platform you are on, the agent needs to outlive the HTTP request that started it.

Concurrency is I/O-bound, not CPU-bound. Hundreds of concurrent runs may need almost no CPU. Size for connections and memory, not cores, and use an async runtime if your language has one.

Retries are dangerous. A load balancer that transparently retries a failed request will re-run an agent that already sent the email. Nothing about a normal retry policy is safe when the unit of work has side effects.

Make the run a job, not a request

The single decision that resolves most of this: an agent run is a background job with an ID, not a synchronous response.

POST /runs returns a run ID immediately. The work happens elsewhere. The client polls for status, or receives a webhook, or streams updates. The HTTP request that started it is over in milliseconds.

This is not novel — it is how every long-running API has worked for decades — but people reach for the synchronous shape first because the demo worked that way.

It is also, notably, the direction the Model Context Protocol has gone. Its Tasks extension lets a server return a durable handle for a long-running request, so clients can poll for status and retrieve the result later. When the protocol standardises around a pattern, it is because everyone implementing it arrived at the same place.

You get four things for free by making runs jobs: they survive deploys, they can be cancelled, they can be inspected while running, and you can rate-limit at the queue instead of the edge.

State, and where it should not be

Agents accumulate state — message history, tool results, intermediate conclusions — and the tempting place to keep it is process memory. Don't.

Hold run state in a store keyed by run ID, and treat the worker as disposable. Then a deploy mid-run is a resumption rather than a loss, and any worker can pick up any run.

This is the same conclusion MCP reached when it went stateless: no connection-scoped sessions, every request self-describing, state held in explicit handles rather than in whoever happens to be holding the socket. Which brings the same warning with it — the spec names state handle hijacking and requires that servers MUST NOT treat possession of a handle as authentication. Your run IDs are exactly this. Make them unguessable, bind them server-side to the authenticated user, and reject them when anyone else presents one. See MCP security.

Cost is a runtime concern, not a monthly surprise

The failure everyone experiences once: an agent loops, and nobody notices until the bill.

Three controls, all cheap, all worth having before the first production run.

Hard caps per run. Maximum iterations, maximum tokens, maximum wall time. Not warnings — stops. A run that hits its cap should fail loudly and leave a trace, which is a vastly better outcome than one that quietly spends forty dollars.

Budgets per tenant. A per-user or per-account ceiling over a window. Without it, one pathological user is an outage for everyone else.

Per-span token accounting. Run totals tell you the bill; the per-step breakdown tells you which tool's verbose output is causing it. See how to trace and debug an AI agent and how to optimise agent cost and latency.

Prompt caching deserves a mention because it is the rare optimisation that is nearly free: agent runs re-send a growing context on every iteration, which is exactly the shape caching is built for. Prompt caching covers it.

What breaks that you did not plan for

Provider errors are normal traffic. Rate limits, overloads and timeouts are not exceptional at volume. Exponential backoff with jitter, and a real decision about what happens when the model is simply unavailable — fail the run, or queue it?

Tool calls fail more than the model does. Third-party APIs go down. The important design choice is what the model sees: a structured, informative error it can reason about, or a stack trace. An agent handed Error: 500 will usually retry the same call forever. An agent handed The billing API is unavailable; this is temporary and not caused by your arguments will often do something sensible.

Non-determinism defeats your reproduction steps. The same input produces a different path. Traces are the only reliable way to reconstruct a specific failure, and they need to be on before it happens.

Deploys land mid-run. Decide explicitly: drain and finish, or cancel and resume. Both are fine; not choosing is not.

Deploy the tools, not just the agent

Easy to forget: an agent is only as available as the things it calls.

If it depends on MCP servers, those are now production dependencies with their own uptime. A local stdio server is a subprocess you must ship, install and version alongside the agent. A remote one is a third-party service whose outage is your outage, and whose OAuth token expiry is your 3am page. MCP transports covers the difference; the deployment consequence is that we use the hosted one is an availability decision, not just a convenience.

Pin versions. npx -y some-server@latest in a production config means an upstream release can change your agent's behaviour without a deploy on your side — which is the same class of problem as an unpinned dependency, with a model in the middle making it harder to spot.

And keep the tool surface small. Every tool description occupies context on every turn, and a model choosing among ninety tools chooses worse than one choosing among twelve. Fewer tools is cheaper and more accurate, which is an unusually easy trade.

Security does not scale down from the demo

Two things change the moment an agent is deployed rather than run locally by its author.

It reads untrusted input. In development, you typed the input. In production, it comes from users, web pages, emails, and tool results — any of which may contain instructions aimed at the model. This is architectural, not fixable by prompting: prompt injection.

It holds credentials that are not yours. A local agent uses your access. A deployed one acts for many users, and the boundary between them is now code you wrote. Scope tokens per user, never share a service credential across tenants, and assume any tool that reads untrusted content might be used to exfiltrate whatever the agent can reach.

The layered controls are in guardrails and safety for AI agents, and the question of where a person sits in the loop is in human-in-the-loop patterns.

A deployment checklist

Before an agent takes real traffic:

• Runs are background jobs with IDs, not synchronous requests
• Run state lives in a store, workers are disposable
• Run IDs are unguessable and bound server-side to a user
• Hard caps on iterations, tokens and wall time — enforced, not advisory
• Per-tenant spend budgets
• Tracing on, with full inputs and outputs and a conversation ID
• Tool failures return structured errors the model can reason about
• Retries are explicit and idempotency-aware — never transparent at the load balancer
• MCP server versions pinned, and their availability treated as your availability
• Tool credentials scoped per user and least-privilege
• Approval required for anything irreversible and external
• A decision on what a mid-run deploy does

Most of that is ordinary distributed-systems discipline. The genuinely agent-specific items are the caps, the traces, and the fact that a retry can send an email twice.

If you are earlier than this — still deciding whether the thing should be an agent at all — agent vs workflow vs single call is the better starting point. A surprising number of production agents should have been workflows, and workflows deploy like ordinary software.