An enterprise AI agent is no longer a simple interface when it can query data, call APIs, create records, send messages, modify states or execute actions. At that moment it is no longer enough to ask if “he responds well.” You have to design what it can do, with what resources, under what conditions and how the organization recovers when it makes a mistake.
Governance is not a bureaucratic layer that is added later. It is part of the architecture. NIST's AI RMF framework is designed precisely to help organizations manage AI system risks during design, development, use, and evaluation, and its profile for Generative AI extends that approach to specific generative system risks.
1. Model risk by action, not by “intelligence level”
The risk of an agent does not depend solely on the model used. It depends on the available actions and the consequences of an error. A mediocre model with permission to delete data can be more dangerous than an excellent model limited to summarizing documents.
For each capacity, evaluate at least four dimensions: impacto, reversibilidad, data sensitivity y frecuencia. Sending an internal draft is different from sending an offer to a client. Consulting a file is different from modifying it. Preparing a payment is different from executing it.
- Low risk: internal search for non-sensitive information, summary, preliminary classification.
- Medium risk: Create tasks, update non-critical fields, prepare responses for review.
- High risk: send external communications, modify critical data, approve operations, act on payments or permits.
2. Define autonomy by steps
There is no single “autonomous or non-autonomous” decision. You can design levels.
- Assistance: the agent proposes, a person executes.
- Execution with approval: The agent prepares the action and waits for human confirmation.
- Limited execution: can act automatically within defined rules and thresholds.
- Supervised autonomy: manages an entire flow, but logs actions, applies limits, and escalates exceptions.
The level should depend on risk, not the desire to automate. A reversible, frequent and well-defined action can be given more autonomy. An irreversibly costly decision should retain human review even if the model is very good.
3. Apply least privilege
The agent should have only the permissions necessary for its task. Don't use a global administrator credential because “it's easier.” Create separate service accounts or scopes when the platform allows it.
If an agent only needs to read calendar, they should not be able to delete events. If you must create drafts, you do not need permission to send. If you query a database, you separate reading from writing. If you need to write, restrict tables, operations, or endpoints.
Separate secrets by function
Don't mix all credentials into a single accessible context. An agent does not need to know the secret that another service uses. Manage secrets outside of the prompt and deliver them only to the connector that needs them.
The best mitigation for a dangerous action is not to ask the model to “be careful.” The thing is that the architecture does not allow it to be executed outside of the rules.
4. Design real human supervision
“Human in the loop” does not mean showing a screen with an approve button. The person needs enough context to detect an error.
An approval should show what action is proposed, on what object, what data justifies it, what consequences it has, and what fields were generated or inferred by AI. If the reviewer has to re-investigate everything from scratch, the supposed savings disappear.
Avoid automation of approval
If a person approves hundreds of identical proposals without reading them, control exists only formally. When the volume makes it unfeasible to review each case, redesign the control: review by samples, thresholds, deterministic rules or separation of risk cases.
5. Evaluate the system with real tasks
A demo does not validate an agent. Build a set of representative cases and difficult cases: incomplete data, contradictions, adversarial inputs, out-of-scope requests, and tool errors.
Define metrics related to the result: extraction accuracy, correct escalation rate, false actions, omissions, resolution time, cost per case and critical errors. For generative tasks, human evaluation is still necessary in dimensions where there is no sufficient deterministic metric.
Testing before and after each change
Changing model, prompt, tool or version can alter behavior. Maintain a regression set. If an update improves average but breaks a critical case, you need to detect it before production.
6. Protects the agent from untrusted instructions
When an agent reads emails, web pages, documents, or messages, that content should be treated as data, not authorized instructions. A document may contain text that attempts to manipulate the agent's behavior.
Clearly separate system instructions, policies, and external content. Limits which tools can be invoked from untrusted input and requires confirmation for sensitive actions.
7. Design safe failures
Systems fail: APIs do not respond, credentials expire, data arrives incomplete, and models return invalid formats. The important question is what happens next.
For high-impact actions, a failure should safely close the flow rather than improvise. If a mandatory piece of information is missing, don't invent it. If an API returns an ambiguous state after executing an action, reconcile before retrying. If there is a risk of duplicating a publication, payment or order, idempotence applies.
Classify errors
- Transients: timeout, time limit, supplier downtime. They can support controlled retry.
- Data: missing fields, invalid formats. They require correction or scaling.
- About politics: action outside permissions or threshold. They must be blocked.
- Ambiguous: We do not know if the action was executed. They require conciliation before repeating.
8. Observability: knowing what you did and why
Records sufficient events to reconstruct a decision: case identifier, flow version, model if relevant, tools invoked, non-sensitive parameters, result, errors and approvals. Do not keep secrets or unnecessary data in logs.
Observability serves for operation and improvement. If the exception rate increases, you need to know at what stage. If the quality changes after an update, you need to be able to compare versions.
9. Dependence on suppliers and continuity
An AI system typically depends on multiple services. Document what happens if a supplier changes price, model, limits, or availability. It doesn't mean building substitutes for everything; It means knowing critical dependencies and designing degradation paths where necessary.
It preserves stable internal formats, abstracts important integrations, and prevents business logic from unnecessarily depending on a proprietary feature that is difficult to replace.
10. Data and privacy
Minimizes data sent to the model. If a task can be resolved with partial fields, do not send the complete file. Reviews contracts, processing regions, retention, and supplier controls based on applicable sensitivity and obligations.
Technical governance is not a substitute for legal advice. When there are personal data, regulated sectors or decisions with relevant effects, it incorporates specific legal and compliance review.
11. Use reference frames without turning them into empty checklists
El NIST AI Risk Management Framework proposes a structured approach to managing risk and trust in AI systems. Its generative AI profile adds specific considerations of generative systems. These frameworks are useful as a reference, but must be translated into concrete controls of the actual process.
The question is not “do we meet a framework?” It is “what risk exists in this case, who owns it, what control reduces it, and how do we know that the control works?”
12. Checklist before giving autonomy to an agent
- Lists all available tools and actions.
- Classify each action by impact and reversibility.
- Reduce permissions to the minimum necessary.
- Defines which actions require approval.
- Design the treatment of untrusted input.
- Create an evaluation set with real and adversarial cases.
- Define spending, volume and frequency limits.
- Implements idempotence for repeatable actions.
- Classify errors and retry policies.
- Design logs without secrets.
- Defines human escalation and operational responsibility.
- Test recovery in case of supplier failure.
- Documents how to quickly disable the agent.
- What is the worst action you can take with your current permissions?
- Can external input induce you to use a sensitive tool?
- What happens if the API confirms an action late?
- Can we reconstruct a decision after an incident?
- Who can stop the system?
- How do we know that an update didn't make a critical case worse?
Conclusion
Useful autonomy is not about giving an agent more tools. It consists of allowing you to act within a designed perimeter: minimum permissions, sufficient data, clear boundaries, observability, evaluation and a safe way out when you don't know what to do.
The greater the potential impact of an action, the less confidence should depend on the model “behaving well” and more on deterministic controls around the model.
En how we work We explain how ProjectCore designs processes, controls and architecture before increasing the autonomy of a system.