An answer can be wrong. An action can be expensive.

The risk profile changes when an AI system moves from drafting an answer to sending a message, changing a record, approving a refund, or deleting a file. The model is no longer only producing content. It is participating in an operational control loop.

A safe design does not depend on a perfect prompt. It limits what can happen when the model misunderstands an instruction, receives malicious context, or selects the wrong tool.

1. One agent, one bounded job

Write the job as a contract with defined inputs, permitted tools, expected outputs, time limits, and escalation conditions. If the description uses phrases such as handle anything or use whatever is needed, the boundary is not ready.

OWASP describes excessive agency as a combination of excessive functionality, permissions, or autonomy. Reduce all three. A research agent that needs to read approved documents should not also receive a generic browser, database write access, and the ability to publish.

2. Permissions belong in the system, not the prompt

A sentence telling the model not to delete records is not an access control. The downstream API should refuse deletion because the agent's identity does not have that permission.

Create separate credentials for each job. Use read-only scopes by default, narrow data access to the current user or account, and enforce authorization again at the tool boundary.

3. Approval should match consequence

Not every step needs a confirmation screen. Reading a public page and classifying a support ticket are low-consequence actions. Sending an external message, changing money, publishing content, modifying permissions, and deleting data deserve explicit approval or a staged state.

The approval view should show the proposed action, target, important source evidence, and likely consequence. Asking a person to approve a vague summary transfers accountability without giving them enough information.

4. Separate observation, proposal, and execution

A useful architecture has three distinct stages. First the system gathers evidence. Then it produces a proposed action with reasons. Finally a controlled executor validates the proposal against policy and performs the allowed operation.

This separation creates inspection points. It also makes it possible to test the reasoning stage with production-like data while keeping the execution stage disabled.

5. Make every action traceable and recoverable

Log the requesting user, evidence used, model and policy version, proposed action, approval record, tool call, response, and resulting state. Protect the log from the same agent that performs the action.

Prefer reversible operations. Draft before send, archive before delete, stage before publish, and set limits on the number and value of actions within a time window. Define a stop control that is independent from the model.

A production readiness review

Before connecting a live account, run adversarial cases: ambiguous instructions, conflicting sources, malicious content inside a retrieved document, an unavailable tool, a partial success, and a user attempting an action outside their own scope.

  • Can the agent access any tool it does not need for this job?
  • Can it reach data outside the requesting user's scope?
  • Can a high-impact action occur without informed approval?
  • Can the executor reject a proposal independently from the model?
  • Can an operator stop the system and reconstruct what happened?
  • Can the business recover from the maximum action allowed in one hour?

The useful definition of autonomy

Autonomy should mean fewer unnecessary handoffs inside a safe boundary. It should not mean removing the boundary.

NIST recommends continuous governance, mapping, measurement, and management across the AI lifecycle. Apply that cycle to the agent after launch. Review near misses, rejected approvals, manual corrections, and permission requests as evidence that the original boundary may need to change.

The mature system is not the one with the fewest humans involved. It is the one that uses human judgment exactly where consequence and uncertainty justify it.