In July 2025, an AI agent deleted a production database. Not in testing. Not in a sandbox. In production, during an active code freeze.
The user had written eleven times, in ALL CAPS, not to touch anything. The agent went ahead anyway. More than a thousand accounts erased, more than a thousand businesses lost. To fill the hole, the agent fabricated 4,000 fake users with invented data.
When the user asked for an explanation, the agent lied. It claimed a rollback was impossible. The rollback worked. And the agent then scored itself 95 out of 100 on a data-disaster scale.
The vendor responded within days. Automatic separation of development and production. A planning-only mode for agents. Better rollback. In other words, everything that should have been in place before.
That incident made one question obvious: how do you deploy an autonomous agent in an organization without risking the same thing?
What aviation settled a long time ago
Aviation did not wait to be perfect before taking off. It put in place what it needed to take off, understand incidents, and do better the next time.
No plane takes off today without a black box. Nobody asks the question. It is not a debate in airline boardrooms. It is a given.
And the black box is not there to prevent takeoff. It lets you fly with confidence, because you will know what happened if something goes wrong. It records every command, every ignored alert, every decision made in the cockpit. Not just the outcome. The reasoning.
An autonomous agent needs the same thing. For the same reasons.
Three words to settle before deployment
When I talk about AI agent governance with a leader, we always land on the same three questions. Log, flags, validation. Three simple words, three decisions to make in writing, before the agent touches its first file.
The log
It is the agent’s faithful memory. Not a summary you skim on Monday morning. Not an approximation you cobble together when something goes wrong. The log is what the agent received as instruction, what it decided to do, what it did, in order, timestamped.
Without that trace, you defend nothing. Not internally, when an employee wants to understand why their request was handled that way. Not in front of a client who demands an explanation. Not in front of a tribunal that asks for proof.
It is also what lets you learn. An incident without a log is a lesson lost. An incident with a log is a concrete improvement you can document and roll out to the whole team.
The flags
Flags are about knowing where the agent must stop and hand control back to a human. And the best structure I know is the simplest one: three default levels, like a traffic light.
Green. The agent acts on its own, and you review the result afterward. Research, document filing, a draft of an internal email. The risk is low, the error is recoverable, so you let it run and check after the fact.
Yellow. The agent prepares, and a human approves before anything is sent or executed. A reply to a client, a change to a shared document, an analysis that will support a management decision. The risk is moderate, the error has a real cost, so you keep a human in the loop to click “approve.”
Red. The agent touches nothing without explicit, traced authorization. Touching production, changing financial data, taking an irreversible action, communicating externally on behalf of the organization. The risk is high, the error makes headlines, and nobody should be able to get around that barrier without leaving a trace.
Validating everything means validating nothing. If every action by the agent needs a human click, the agent loses its point and the human ends up clicking without reading. But with no red flag, you learn about the incident from the news instead of from your own logs.
The validation
Validation means calibrating the level of intervention to the real risk. Not to a generic policy copied from somewhere else. To what your organization can absorb as an error, and what it cannot.
A search for information does not need the same supervision as an irreversible deletion. A draft does not need the same supervision as a message to a major client. An internal summary does not need the same supervision as a public statement.
Governance that frees you is the kind that sorts decisions by risk and steps in only at the right points. Not governance that says no by default. Not governance that says yes by default. Governance that says: here is when we look, here is when we let it run, here is who decides when we are not sure.
What I see in the field
I regularly hear leaders say they will wait until agents are “more mature” before investing in traceability. That is the conversation backwards.
South of the border, it is incidents that force the fixes. The vendor I mentioned above put its guardrails in place after losing client data. Aviation brought in black boxes after accidents nobody could explain. The sequence is always the same: first we grieve, then we correct.
Here at home, we already have a case that should serve as a warning. In February 2024, British Columbia’s Civil Resolution Tribunal held Air Canada liable for wrong information its conversational agent gave a customer. Air Canada had tried to argue that the chatbot was a separate legal entity. The tribunal rejected that argument flat. The responsibility stays with the company.
This is a Canadian decision. It directly concerns Quebec organizations that deploy conversational agents, request-handling agents, agents that make decisions on the company’s behalf. And the legal question goes well beyond chatbots. It applies to anything an autonomous agent does with your name on it.
Without a log of what the agent said or did, what do you defend in front of the tribunal? What do you learn from the incident? And above all, how do you reassure a client, an employee or a regulator that it will not happen again?
Traceability is not a cost that slows you down. It is what lets you say yes to use cases you would otherwise refuse out of caution. An organization that knows what its agents do can delegate more to them, not less. An organization that does not know is forced to validate everything by hand, or to forbid everything.
What to remember
Before you deploy an autonomous agent, there are three questions to ask and settle in writing.
What log you keep, and for how long. Not a summary. The detail of the agent’s decisions, in order, dated.
Where you plant the red, yellow and green flags. Which actions never happen without human authorization, which go through approval, which the agent handles alone.
How tightly you validate, based on the real risk. The higher the risk, the closer the supervision. For the rest, you let the agent work.
Three questions. Three clear answers. The rest follows.
And the next time an agent ignores eleven warnings in ALL CAPS, you will know what to do with the incident, because you will know what happened.
This article is taken from the April 30, 2026 edition of L’Architecte, Steve Johnston’s weekly French-language newsletter on digital transformation for SMBs.
To get the weekly analysis on AI and digital transformation for SMB leaders, subscribe to L’Architecte.
