In two years of shipping AI into hospitals, law firms, and insurance carriers, I’ve watched almost every production incident stem from a variant of the same failure mode: nobody at the vendor knew the AI was wrong until someone outside the company told them. So I built a simple pre-flight check, two questions we run before any deployment goes live. If either answer is “we don’t know,” we don’t ship. Here’s why those two questions matter more than any test suite.
Every AI production incident I’ve watched at an enterprise customer over the last two years traces back to the same failure mode.
Nobody at the vendor knew the AI was wrong until someone outside the company told them.
Sometimes that “someone outside” was a regulator. Sometimes it was a very unhappy patient, client, or claim filer. Sometimes it was a competitor who noticed something in the output before the vendor did. The specifics were different every time. The failure mode was identical.
The AI was wrong. Nobody at the vendor knew.
That failure mode has a specific structure. It’s not about the AI being smart or dumb. It’s not about the eval suite being good or bad. It’s about whether the vendor has an observation system paired with the AI, and whether that observation system has a named human attached to it.
Most vendors have neither. They have monitoring dashboards. They have alerting infrastructure. What they don’t have is a specific answer to two very specific questions.
Those two questions are the pre-flight check I run on every Attri deployment. If either answer is “we don’t know,” the deployment doesn’t ship.
Here they are.
Question 1: If this decision was wrong, how would we know within an hour?
Not “eventually.” Not “within a day.” Not “when a customer complains.”
Within an hour.
That specific time bound forces a very concrete answer. It’s not a philosophical question. It’s an operational one. What is the signal? Where does it live? Who watches it? What’s the alert threshold? Who gets paged, on what schedule, with what runbook?
Most teams cannot answer this question honestly for their production deployments. That’s not a criticism, it’s just a fact. And it’s precisely why customer-detected failures are the norm in enterprise AI, rather than the exception.
If your honest answer is “we don’t know,” you don’t ship. Because “we don’t know” here means “we won’t know until the customer tells us.” And by then the incident has already happened.
If your only detection system for AI errors is “the customer will complain,” you haven’t built a deployment. You’ve built a slow-motion incident.
Question 2: If this decision was wrong, who is the named human that catches it, and how do they get notified?
Named. Not “the on-call team.” A specific human. With a name.
The reason: at 3am on a Sunday, “the on-call team” is one specific person, whether that’s in your incident response doc or not. That person needs to have the authority to pause the agent, the context to understand the incident, and the escalation path to the customer’s compliance officer already in their contacts.
If your incident response document says “engineer will investigate,” you don’t have an incident response document. You have a plan to have an incident.
The named-human requirement forces a discipline that vague plans skip: someone at Attri, and someone at the customer, both have to agree, in writing, that this specific person is the one who catches this specific class of error. That agreement is what makes the deployment defensible, not to us, not to the customer, but to the regulator or auditor who asks “who was responsible?” six months later. Every AI production incident I’ve watched at a customer traces back to the same thing: nobody at the vendor knew the AI was wrong until someone outside the company told them.
Every AI production incident I’ve watched at a customer traces back to the same thing: nobody at the vendor knew the AI was wrong until someone outside the company told them.
Why these two questions, and not three, or five, or twenty?
Because a check with twenty questions is not a check. It’s a checkbox exercise.
Two questions is what you can actually run before every deployment. Two questions is what your engineering team will remember without reading the doc. Two questions is what your customer’s compliance officer can ask you to walk them through on a 15-minute call, and know within those 15 minutes whether they trust the deployment.
Every additional question dilutes the discipline. Two questions is the number that survives the pressure of a launch week.
What this actually looks like at Attri
Before any deployment goes live, my team walks through both questions in a 30-minute meeting. Not a lot of ceremony. Just:
Show me the specific detection signal for wrong decisions. Show me the alert. Show me who gets paged.
Show me the named human. Show me their contact info in the customer’s incident response document. Show me the escalation path.
If both answers are clean, we ship. If either one is fuzzy, we don’t. We fix the fuzzy one and revisit.
Nothing about the technical evaluation. Nothing about the model choice. Nothing about the eval suite, which we’ve already run separately, in a different meeting, with different criteria.
Just those two operational questions.
That 30-minute meeting has done more to prevent customer-facing incidents than any technical work my engineering team has ever shipped. And it’s the meeting my engineering team hates the least, because it takes 30 minutes and it prevents the 2am pages nobody wants.
The Builder’s Takeaways
1. Add a pre-flight check to every deployment. Keep it short. Two questions minimum. Get the answers in writing, in a document the customer signs off on. If either answer is “we don’t know,” fix that before you ship, not after. The version of your team that fixes it before ship-day is 10x cheaper than the version that fixes it in a war room three months later.
2. Name a specific human. Every time. Not a team. Not a role. A person, with a phone number, with the authority to pause the agent, with the context to understand what happened. If you can’t name that person, you haven’t finished building the deployment. Every deployment we’ve renewed at Attri has that name written in the contract appendix. Every deployment we’ve lost eventually turned out to be missing it.
3. Optimize your monitoring for the customer not noticing. Every dashboard, every alert, every log should be built so that when something goes wrong, you see it first, with enough time to fix it before the customer notices. That gap between when you know and when the customer knows is the entire product of enterprise AI trust. Everything else is table stakes.
Final statement
The AI industry loves to talk about model capability, benchmark scores, and eval suites. Those all matter. But none of them are what determines whether your customer renews.
What determines whether your customer renews is whether they trust that you catch problems before they do.
Two questions is what builds that trust.
Two questions is what has separated the enterprise AI deployments I’ve watched succeed from the ones I’ve watched fail, and the pattern is consistent enough that I’d bet the rest of the decade on it.
Two questions is the smallest thing that actually works.