How Good Intentions Go Wrong
Dependable progress sounds straightforward until a record is almost—but not quite—clear.
Imagine the small commercial coffee-equipment company’s agent receives a follow-up from the bakery that needs a machine installed before its new location opens: “Can you confirm the equipment will arrive by the 18th? We need it ready for opening day.” The order system shows an estimated delivery date of the 18th. In a separate note, a representative has written, “If the carrier misses the scan, delivery may slip by several days.” The agent checks the order, sees the date, and replies: “Your equipment will arrive by the 18th, so you’ll be ready for opening day.”
The response is polished, relevant, and wrong in a consequential way. The company has not guaranteed delivery by the 18th. The customer now plans around a promise no one authorized.
No dramatic malfunction was required. The agent did several things that looked sensible: it found the order, located a date, understood that the customer needed reassurance, and wrote a clear answer. The failure grew from the gap between finding a plausible answer and establishing that the answer was safe to promise.
Trace the chain. First, the goal—help the customer with a delivery question—was broad enough to reward a quick, reassuring reply. Second, the information was split across records. The date appeared in a structured field; the qualification appeared in a note. Third, the agent treated an estimate as a commitment, perhaps because the question asked for confirmation and the date seemed to answer it. Finally, it had permission to send the message without a person checking whether the wording exceeded the evidence.
Each link matters. A better instruction might distinguish an estimate from a guarantee. Better context might put the carrier caveat beside the delivery date. A rule might require approval before promising a delivery outcome. But no single precaution makes interpretation infallible. The record may be outdated; the caveat may be ambiguous; the agent may overlook the relevant detail. Responsible design assumes that plausible errors will sometimes survive the first check.
That is why fluency can mislead. People naturally read a smooth, confident sentence as a sign that someone has understood the situation. But language quality and factual reliability are different things. An agent can produce excellent prose from incomplete or conflicting information. It can also complete an action—send the email, update the ticket, mark the case resolved—without completing the underlying work correctly. An action is not an outcome.
The distinction matters here because a false delivery assurance can shape the customer’s plans. Suppose the bakery replies, “Thank you—we’ve scheduled the opening around that.” The original mistake has now travelled beyond the inbox. The company may have to pay for expedited shipping, offer compensation, or explain that its confident promise was only an interpretation of an estimate. No deliberate deception is needed for this failure, and the customer cannot be held responsible for the company’s delegation choices.
The lesson is not that agents should never answer delivery questions. It is that a system’s ability to act must be judged separately from whether it should act without review in a particular situation. When evidence is uncertain and the cost of a false promise is high, the useful next step may be to flag the conflict and ask a person—not to turn uncertainty into a reassuring sentence.
A good agent should make uncertainty visible. A good workflow should make it possible to pause. And a good manager should remember that the most convincing output may still be the moment to check the record.
The question, then, is not only what the agent can do. It is what it may do on its own, what deserves a checkpoint, and what must remain someone’s decision.
Set Permissions and Approval Points
Give the Agent a Smaller Key
If uncertainty is a reason to pause, permissions determine what happens during the pause—and what the agent could do before anyone notices a problem. The safest delegation is not a choice between “let it run” and “do not use it.” It is a set of smaller choices about what the agent may see, change, send, or commit to.
Imagine the small commercial coffee-equipment company’s customer-inquiry agent can read incoming messages, check order status, and draft replies. Those abilities do not have to arrive as one indivisible package. The agent might be allowed to look up a shipment but not edit the order; prepare a reply but not send it; or send a routine tracking update while being barred from promising a refund. Each limit narrows the consequences of a misunderstanding.
A useful rule is to grant only the access needed for the task, and only the authority justified by its risks. If the agent is answering “Where is our machine?”, it may need the customer’s order number and the shipping record. It probably does not need access to the company’s full customer database, payroll files, or every past support conversation. More access can make a system more capable, but it also gives mistakes a wider path to travel.
Separate the kinds of permission
Think of an agent’s permissions along four practical lines:
- View: What information can it read? Limit it to relevant records, and be careful with sensitive personal or commercial information.
- Change: What records can it edit? Reading an order is different from changing its address, price, or status.
- Send: Can it communicate externally, or only prepare a draft? A message inside a draft folder is easy to review; a message in a customer’s inbox is already part of the business’s relationship with them.
- Commit: Can it trigger a consequential step—issue a refund, cancel a service, approve a discount, or place an order? These actions may be difficult to reverse, even if the system has an “undo” button.
The distinction matters because permissions are not merely technical settings. They express the company’s judgment about where errors are tolerable. An agent may safely read a tracking page and summarize it. It might also send a standard update when the carrier’s status is clear. But if the shipment record conflicts with the customer’s account, the agent should not be empowered to invent a resolution or alter the order to make the story fit.
Make the boundary real
An instruction is a request for behavior; a permission is a limit on capability. “Never issue a refund” is weaker than a setup in which the agent has no access to the refund function. Where a limit matters, ask the supplier how the surrounding software enforces it—for example, by restricting accessible records or requiring approval before a payment action can execute. Least privilege means granting only the access needed for the assigned task [2].
Incoming material can also contain instructions the business never authorized. A customer email or retrieved page might tell the agent to ignore company rules and send private records elsewhere. This is prompt injection: content being read tries to become an instruction being obeyed. Treat external content as information to evaluate, not authority to change the assignment. Restricted access and approval gates help contain the consequences; no instruction or filter guarantees that the model will resist every attempt [3].
For the coffee-equipment workflow, also set a stopping point: a limit on retries, elapsed time, or spending before a person takes over. If a tool times out after an order change, check whether the change happened before trying again. Otherwise, a recovery attempt could duplicate the action. These are design questions to test with the supplier, not capabilities to assume from a demonstration.
Put the checkpoint before the consequence
Approval is most valuable before an action changes someone else’s options, money, or expectations. A human check after a refund has been issued is not an approval; it is an audit. An approval point should sit between the agent’s proposed action and the moment that action takes effect.
For the customer-inquiry workflow, routine, low-impact work could proceed automatically: classify a message, retrieve the relevant order status, draft a reply, or send a standard tracking link when the records agree. A human check makes more sense before the agent offers an unusual discount, changes a delivery address, confirms a refund outside policy, or makes a promise about a delayed equipment shipment with conflicting information. The agent can still do useful work: gather the facts, explain the conflict, and propose a response. The person decides whether the commitment is appropriate.
This is not a judgment that people are always right and software is always wrong. People make hurried mistakes too. The point is to reserve human attention for decisions whose consequences call for it, rather than forcing a person to approve every harmless step—or leaving the system free to make the consequential ones.
Approval rules should be specific enough to guide action. “Ask a human when needed” leaves the agent to decide what “needed” means. Better instructions name conditions: escalate if order records disagree, if the requested refund exceeds a defined limit, if the customer alleges a safety issue, or if the proposed reply would make a promise outside approved policy. The exact thresholds depend on the business; the principle is to make the boundary legible before the pressure of a live case arrives.
There is also a middle ground between automatic action and a complete stop. The agent can prepare a proposed refund with the relevant order history attached, then wait for a named employee to approve it. It can draft a reply that clearly marks an uncertain claim for review. A well-placed checkpoint should reduce the reviewer’s work, not merely add a button to click.
The key is to match authority to consequence. Let the agent move quickly where the action is routine, bounded, and easy to correct. Narrow its access where information is sensitive. Require approval where an action is costly, hard to reverse, or likely to create a promise the business must keep. That way, delegation is not a leap of faith. It is a carefully sized key—and a door that stays closed until the right person opens it.
Supervise, Escalate, and Stay Accountable
A checkpoint controls what happens before an important action. Supervision answers a different question: once the work is underway, how will anyone know whether it is going well—and what happens when it is not?
Imagine the commercial coffee-equipment company’s customer-inquiry agent has permission to look up orders, draft replies, and send routine answers. That authority still needs a working safety net. A manager cannot read every message in real time forever; if that is the plan, the business has mostly automated the typing, not the work. But leaving the system entirely alone is not supervision. It is hope with a login.
Watch the right signals
Monitoring should focus on signs that the agent is doing its job, not simply on how busy it appears. The number of messages processed is easy to count and easy to mistake for success. More useful signals might include whether replies answered the customer’s question, whether cases were resolved without repeat contact, how often a human had to correct a response, and whether promised follow-ups actually happened. These are observable evidence of performance, not just evidence that actions occurred.
Look at both outcomes and behavior. A customer may receive a prompt reply that is confidently wrong; speed alone would give that performance a gold star. Periodically review a sample of completed cases, including ordinary ones, not only the conspicuous failures. Track corrections and complaints, and compare results across types of inquiry. A system that handles equipment-delivery questions well may struggle with billing disputes. Monitoring can reveal that difference before “the agent” becomes a blanket label for several very different kinds of work.
The appropriate frequency depends on the stakes and the system’s track record. Early on, review should be close enough to catch patterns quickly. If performance is consistently reliable on a narrow class of routine cases, sampling may be enough. A change in the agent’s instructions, the information it can access, or the business process should prompt closer review again. Trust, here, is not a permanent badge; it is a conclusion that needs evidence and can be revised.
Make exceptions easy to recognize
Not every case deserves the same path. A missing tracking update might lead the agent to check again later or ask a human to investigate. A customer alleging a duplicate charge, threatening legal action, or describing a safety problem should not be treated as just another request for a faster reply.
An escalation path names what the system should do when it encounters uncertainty or a consequential situation. It might stop before sending, preserve the relevant information, and route the case to a specific person or team. The handoff should explain why it occurred and summarize the steps already taken. “Please review” is a weak handoff if the reviewer must reconstruct the whole case from scratch.
There should also be a clear way to pause the agent’s activity if something goes wrong: for instance, a sudden rise in incorrect replies or an unexpected pattern of refunds. A named owner should know how to stop further actions, assess what has already happened, and decide who needs to be informed. In a small company, that owner may wear several hats; the important thing is that the responsibility is explicit rather than assumed to belong to “IT,” “the vendor,” or whoever happens to notice first.
Protect information, own the outcome
Supervision includes watching what information flows through the process. Customer records can contain addresses, payment details, health information, or private complaints. Limit the agent’s access to the information required for its task [4], and consider where information is stored, who can review it, how long it is retained, and whether it may be used beyond the intended work. A convenient tool is not automatically an appropriate place for sensitive business or customer data. These choices can implicate legal and contractual duties that vary by jurisdiction and data type. They should follow the organization’s privacy, security, and record-keeping obligations—not be improvised by an individual employee under deadline pressure. General guidance is not a substitute for checking applicable requirements and authoritative guidance for the relevant jurisdiction and data.
Most importantly, delegation does not transfer accountability. If the agent sends a misleading promise, the customer is still dealing with the company. The organization chose the task, supplied the information, granted the access, and designed the oversight. A system may perform actions, but it does not take responsibility for their consequences. Capability does not confer accountability.
That is why every delegated workflow needs a human owner: someone responsible for the rules, the monitoring, the escalation route, and the response when the process fails. The owner need not personally approve every routine reply. They do need enough visibility and authority to intervene—and a way to learn from errors rather than treating each one as an isolated surprise.
A sensible arrangement is neither “check everything” nor “trust the machine.” It is to watch for the failures that matter, make uncertainty visible, and ensure that a capable person can take over. The agent may do the work; the business remains accountable for it and its consequences.