Guardrails and Safety for AI Agents

Why agents need more than a chatbot's safety filter, and how to layer input validation, scoping, approval, and sandboxing.

A chatbot that gives a bad answer wastes your time. An agent that gives a bad answer can delete a repository, send an email, or charge a card. That difference is the entire reason guardrails for agents look nothing like content moderation for chat.

If you haven't yet, read how to build your first AI agent for the baseline loop this article assumes — an LLM that plans, calls tools, and observes results. Here we cover what keeps that loop from doing damage.

Why agents are a different risk category

A chatbot's output is text a human reads before acting on it. An agent's output is often an action executed directly — a file write, an API call, a payment, a message sent on your behalf. The human-review step that used to catch mistakes is gone by default, which means the model's judgment is now a control surface, not just a UX layer.

Agents also consume untrusted content as part of their reasoning: search results, scraped pages, API responses, email bodies, PDFs a user uploads. Any of that content can contain instructions, and the model doesn't reliably distinguish instructions from my developer from instructions embedded in a page I just fetched. That's the mechanism behind prompt injection, which OWASP ranks as the top risk for LLM applications for two editions running — see the OWASP Top 10 for LLM Applications and the OWASP AI Agent Security Cheat Sheet, which extends this to multi-step, tool-using agents.

The threat models

Design guardrails against specific failure modes, not a vague sense of safety:

  • Prompt injection via tool results or retrieved content. Text pulled from a webpage, document, or API response tells the model to ignore its instructions, exfiltrate data, or call a different tool than intended. This is indirect injection — the attacker never talks to your agent directly, they poison something it will read.
  • Tool misuse. The model calls a real, legitimate tool in a way you didn't intend — deleting instead of archiving, messaging the wrong recipient, running a destructive query — because its plan went wrong or it was manipulated.
  • Data exfiltration. The agent has read access to sensitive data and a channel that can leave your system (send email, post to a URL, write a public file), and something convinces it to combine the two.
  • Runaway loops. The agent retries, re-plans, or spawns sub-tasks indefinitely — burning API spend, hammering a rate-limited service, or repeating a harmful action before anyone notices.

Each layer below maps to one or more of these.

Layer 1: Input validation

Treat everything that enters the context window as untrusted unless it came from your own system prompt: user input, tool outputs, retrieved documents, other agents' messages. Practical controls:

  • Strip or flag content that looks like embedded instructions (ignore previous instructions, role markers, suspicious formatting) before it reaches the model.
  • Constrain and type-check tool inputs and outputs against a schema rather than passing raw strings through.
  • Segregate instructions from data structurally where your framework allows it (e.g., distinct message roles or delimited blocks), so the model has a better signal for what's authoritative.

Input validation reduces the injection surface but doesn't eliminate it — no filter catches every phrasing. It's a mitigation layer, not a fix.

Layer 2: Output filtering and validation

Before an agent's output becomes an action or is shown to a user, check it:

  • Validate structured outputs (tool calls, JSON) against a strict schema and reject malformed or unexpected calls rather than best-effort parsing them.
  • Scan for secrets, PII, or credentials the model shouldn't be emitting, especially in logs or messages sent externally.
  • For anything user-facing, apply the same content checks you'd use on any generated text.

Layer 3: Least-privilege tool scoping

This is the single highest-leverage guardrail, and it belongs at design time, not as an afterthought. The same instinct that makes you scope a database user's permissions applies to agent tools — see how to design tools your AI agent can actually use for the design side of this.

Concretely:

  • Give each tool the narrowest capability that accomplishes its job. A search tickets tool shouldn't also be able to close them.
  • Scope credentials per tool and per session — short-lived, audience-bound tokens beat long-lived API keys shared across the whole agent.
  • Separate read from write. Let an agent read freely where the data isn't sensitive; gate every write and delete.
  • Cap blast radius: a tool that can modify records should only be able to touch the records the current task legitimately needs, not the whole table.

A simple policy might look like this:

tools:
  search_knowledge_base:
    access: read-only
    scope: public_docs
    approval: none

  send_email:
    access: write
    scope: outbound
    approval: required        # human confirms recipient + body
    rate_limit: 10/hour

  delete_record:
    access: write
    scope: current_workspace
    approval: required
    reversible: false         # forces approval regardless of other rules

OWASP calls the failure mode Excessive Agency — an agent with more functionality, permission, or autonomy than its task needs. The fix is the same principle applied concretely: fewer tools, narrower scopes, shorter-lived credentials.

Layer 4: Human-in-the-loop approval

Not every action needs a human. Reviewing every step defeats the purpose of automation. The design question is which actions are irreversible or high-impact enough to require sign-off before execution — sending money, deleting data, publishing content, messaging external parties.

A minimal approval gate:

IRREVERSIBLE_ACTIONS = {"delete_record", "send_payment", "publish_post"}

def execute_tool_call(call):
    if call.name in IRREVERSIBLE_ACTIONS or call.estimated_impact == "high":
        approved = request_human_approval(call)
        if not approved:
            return ToolResult(status="rejected")
    return run(call)

Set the threshold on reversibility and impact, not on how risky an action sounds in the abstract — a bulk delete on a scratch table is lower stakes than a single email to a customer list.

Layer 5: Sandboxing and isolation

Run agent-executed code and risky tools in an isolated environment: containers with no network access by default, ephemeral filesystems, credentials separate from production. If an agent is compromised via injection, sandboxing decides whether the blast radius is one throwaway container or your production database. This is also where multi-agent systems benefit from isolating each sub-agent's tool access rather than sharing one broad credential set.

Layer 6: Rate, spend, and iteration limits

Runaway loops are a distinct failure mode from malicious misuse — the agent can go wrong entirely on its own. Guard against it with hard ceilings:

  • Max iterations or max tool calls per task, with a forced stop and human handoff on breach.
  • Per-session and per-day spend caps on token usage and any paid tool calls.
  • Rate limits per tool, independent of the agent's own judgment about pacing.
  • Timeouts on individual tool calls so a hanging dependency doesn't stall the loop indefinitely.

Monitoring, audit logs, and observability

Guardrails that fail silently aren't guardrails. Log every tool call with its inputs, outputs, and whether it was approved, denied, or auto-executed. At minimum, capture:

  • The full reasoning trace or a summary of it, not just the final action.
  • Which policy or approval rule fired (or should have fired and didn't).
  • Anomalies: repeated failures, unusual tool sequences, spend spikes.

This log is what lets you detect an injection attempt after the fact, tune your approval thresholds, and answer what did the agent actually do when something goes wrong — which it eventually will.

Designing for safe failure

Assume a guardrail will eventually be bypassed or misconfigured, and design containment for that case:

  • Default to denying unrecognized tool calls rather than passing them through.
  • Make destructive actions require confirmation even from trusted internal callers, not just external input.
  • Build a kill switch — a way to halt an agent or an entire class of agents immediately, not just disable new sessions.
  • Prefer reversible actions (soft delete, draft-then-publish) over irreversible ones wherever the workflow allows it, so a bad decision has a recovery path.

Putting it together

None of these layers is sufficient alone. Input validation misses novel injection phrasing; output filtering misses semantically valid but wrong actions; scoping limits damage but doesn't prevent misuse within scope; approval gates only work if the thresholds are right. They're meant to overlap, so a failure in one is caught by another.

Start from least privilege in your tool design — it pays off no matter what else goes wrong — then add approval gates for anything irreversible, sandboxing for anything that executes code or touches production, and logging so you can see what actually happened. For the full picture of where these guardrails fit in an agent's architecture, see how to build your first AI agent.

← Back to Articles