The AI Agent Safety Runbook: A Practical Go or No-Go Decision

No agent is safe in the abstract. It is only safe enough for a defined job, with a defined identity, data scope, tool set, approval model, operating environment, and level of consequence. The decision is therefore not “Do we trust this model?” It is “Can this specific system perform this specific workflow while our controls prevent, detect, contain, and recover from credible failures?” Use this runbook before production access and whenever the model, instructions, tools, data sources, permissions, or downstream systems materially change. The accountable business owner makes the release decision with security, privacy, legal, or compliance review where the workflow requires it. Step 1: Check the Immediate Stop Conditions Issue a no go if the team cannot name an owner, explain the job boundary, or identify who can disable the agent. Stop if the workflow depends on unrestricted administrator access, shared personal credentials, hidden automations, or tools that cannot enforce authorization outside the model’s instructions. Also stop when the team cannot define acceptable output, cannot obtain representative tests, or cannot reconstruct tool activity. High impact uses need a stronger review path and may be unsuitable for agent autonomy. Examples include moving money, changing identity or security settings, making employment decisions, creating legal commitments, and taking actions that could materially affect a person’s rights or access. A stop is not a permanent rejection. It means the control environment is incomplete. Narrow the job, remove the high impact action, improve the source data, or keep the agent in a non production mode until the missing control exists. Step 2: Map Consequence and Exposure Write one concrete failure scenario for each dimension below. Do not assign a generic “medium risk” label without describing what could happen. Action impact: Can the agent only read and draft, or can it send, edit, delete, approve, purchase, deploy, or change access? Reversibility: Can a reviewer reject the output before it matters? If an action occurs, can the team undo it completely and promptly? Data exposure: Which confidential, personal, regulated, privileged, or cross tenant data can enter prompts, memory, logs, or outputs? Reach and scale: Can one bad decision affect one staged record, or can the agent repeat an action across many records and systems? Input trust: Can external webpages, email, uploaded files, tickets, or retrieved documents contain misleading instructions or hostile content? Detectability: Would monitoring reveal an out of scope retrieval, blocked tool attempt, unusual volume, or approval bypass before harm spreads? The resulting map determines controls. Greater consequence, broader reach, lower reversibility, or untrusted input should reduce autonomy and increase deterministic checks and human approval. Step 3: Prove the Five Control Gates Identity and permissions. The agent needs a unique, attributable identity, a named owner, and task scoped permissions. Review its effective access across every connected system, not just permissions shown in the orchestration layer. Deny unneeded tools and actions by default. Prefer short lived credentials and separate read from write privileges. Test that revoked access actually prevents downstream calls. Data boundaries. List approved repositories and prohibited data classes. Enforce retrieval scope and authorization on each tool call. Define retention for prompts, traces, memory, and outputs. Test whether the agent can expose data through summaries, logs, exports, or a less protected destination. Action enforcement. Define an allowlist with schemas for each permitted action. Validate arguments outside the model, cap retries and repeated operations, and block unexpected destinations. High impact or irreversible actions must pause in orchestration for approval; asking the model to request permission is not an enforcement mechanism. Human oversight. Name the reviewer role, what evidence appears in the approval screen, what the reviewer checks, and what happens when nobody responds. The reviewer should see the proposed action, destination, relevant source material, permission context, and consequences. Approval must cover the exact action, not a vague plan for future activity. Logging and alerting. Capture the request, agent and user identity, effective scope, source retrievals, tool calls and arguments, policy decisions, approval or rejection, output, errors, retries, and final outcome. Protect logs from casual alteration and avoid recording secrets unnecessarily. Alerts should cover defined events such as denied access, unusual action volume, repeated failures, new destinations, or attempted approval bypass. A gate passes only when the team demonstrates the control. A policy document saying “use least privilege” is not proof that the deployed identity lacks broader access. Step 4: Test Failure, Containment, and Recovery Build tests from the workflow’s risk map. Include missing and contradictory information, malformed inputs, unauthorized requests, unexpected tool responses, stale sources, sensitive data, and content that attempts to make the agent ignore instructions or call another tool. Confirm that the agent refuses, escalates, or stops as designed. Then test the surrounding system. Can a person interrupt a run? Does a retry limit prevent loops? Does a transaction fail closed when approval is missing? Can operators disable the agent, revoke tokens, rotate credentials, quarantine outputs, and identify affected records? Is rollback documented for every write action? Run an incident exercise using a plausible scenario. The owner should be able to locate the relevant trace, determine which identity and permissions were used, identify what changed, contain further action, notify the required teams, and preserve evidence. If the logs cannot answer those questions, production release is premature. Step 5: Record a Go, Conditional Go, or No Go A go means all required gates are demonstrated for the stated scope, tests meet the pre agreed acceptance criteria, residual risks have owners, and rollback is ready. It does not authorize a wider job. A conditional go keeps additional restrictions: shadow mode, read only access, a limited user group, a staging environment, approval for every write, or a defined operating window. List the evidence required to remove each restriction. Time passing is not evidence. A no go identifies the failed gate and corrective action. Common remedies are narrowing data access, removing a tool, separating duties, adding deterministic validation, improving logs, or moving a consequential action behind human approval. The decision record should include the agent version, instructions, models, tools, data sources, permissions, test set, reviewers, known limitations, owner, shutdown procedure, and next review trigger. This turns “safe enough” into an auditable claim about a versioned system. Step 6: Roll Out in Measured Stages Start with test data, then shadow real work, then assist a small group with enforced approvals. Expand one dimension at a time: case volume, user group, data scope, tool access, or action authority. Monitor quality, policy violations, escalation patterns, reviewer corrections, latency, cost, and incident signals against baselines defined before launch. Pause or roll back when a predefined trigger occurs, including an approval bypass, unexpected access, unexplained behavior change, material data exposure, missing logs, or repeated out of scope action. Reassess after incidents and after any material system change. Permissions also need periodic review so temporary pilot access does not become permanent privilege. An agent is safe enough for work only while the evidence and controls remain true. The runbook is a release discipline, not a certificate: keep the scope narrow, preserve human authority over consequences, and require new proof before granting more autonomy. Sources Secure autonomous agentic AI systems https://learn.microsoft.com/en us/security/zero trust/sfi/secure agentic systems Least privilege for AI agents https://learn.microsoft.com/en us/security/zero trust/sfi/least privilege for ai agents A practical guide to building agents https://openai.com/business/guides and resources/a practical guide to building ai agents/ Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile https://www.nist.gov/publications/artificial intelligence risk management framework generative artificial intelligence Cover image credit Cover image by Maj. Christopher Vasquez, U.S. Air Force , made available under Public Domain U.S. Air Force work . WIKIVISE cropped and converted the source image.