Walk into most boardrooms and the AI conversation is about models: which one, build or buy, how fast can we roll it out. Reasonable questions, but a fraction of the problem. Boston Consulting Group puts numbers on it: Across hundreds of companies, about 10% of the value from AI comes from the algorithm, 20% from technology and data, and 70% from the people and processes around it. The model everyone argues about is the easy part.

You see the problem after the pilot. Capability is everywhere – a team can summarize a contract or stand up a prototype in an afternoon – but little of it survives production, which answers to everyone: What data can it touch, whose permissions does it inherit, who signs off when it's wrong? For the first time in a major technology cycle, access isn't the constraint. Operational readiness is.

‘Trust’ is a trap

The industry has settled on "trust" as the goal – trustworthy AI, responsible AI. I'd push back on that.

Trust is a human thing. It’s what you extend to something that has its own interests and could act against yours. A language model has no such interior life, and worse, it's built to be agreeable: trained to be helpful and deferential (sycophantic). It tells you what you want to hear, says it with confidence, and folds the moment you push. A fluent model earns misplaced trust easily. A careful one never feels trustworthy no matter how good it is.

The useful question is whether you've built the system around the model so that trust is beside the point. Three properties do that work:

• Visibility: can you see what the agent is doing, step by step – the actual actions and data, not its own summary? 
• Predictability: given the same situation, does it do the same thing? 
• Control: can you direct it, bound it, and stop it, and have those limits hold even when the model's eagerness to help pushes against them?

None of the three requires trusting the model.

Predictability has a precondition most teams miss. An agent acts only on what's been made explicit; it can't absorb your norms by watching the way a new hire does. When a customer-facing agent meets a pricing exception no one codified, it doesn't stop: it improvises, and drifts off your goals. Undocumented judgment leads to undefined behavior. 

Governance is a design problem

Because of my background in design, I've watched governance get filed as a technical problem. It's a design problem too. The most capable AI system on earth is worthless if the person using it can't tell what it did, can't anticipate what it'll do, or doesn't know when to step in. Most enterprise AI conversations dwell on infrastructure, models, and architecture. Far fewer dwell on the experience around them, which is where visibility, predictability, and control either become real or stay on a slide.

Provenance has to be legible at the moment of decision, not buried in a log. The system has to signal when it's inside its competence and when it's improvising, so the user knows how much weight an output can bear. And it has to make plain where human judgment is still required, with a real way to intervene. Transparency, accountability, and oversight live in the interface, or they don't live at all.

As AI moves closer to business-critical decisions, that design layer decides whether people can govern the system or merely hope it behaves. Transparency, accountability, and oversight live in the interface, or they don't live at all.

The question everyone avoids

Moonshots are sexy but the projects that create value start small like an app that reads invoices and routes discrepancies to a person, one that flags network disruptions in real time. When the problem is narrow and the judgment is clear, you can build solutions in days, not months. But the speed is earned as your teams learn how to build judgement into the system.

"How do we keep a human in the loop?" has become a reflexive response but without strategy. Having a human in every loop doesn't scale risks turning every role into a glorified babysitter.

A cleaner approach is to map each task by the cost of an error and by whether it runs on explicit data or hard-to-codify tacit knowledge. Low cost and explicit data let AI run unattended. High cost and/or heavy judgment required keep a person in charge.

I'd add a third axis rarely considered: Reversibility. A wrong answer you can roll back tomorrow is a different animal from one that's cheap but permanent like a mispriced quote already sent or a deleted record. Agents make reversibility crucial because they act before anyone sees.

The real differentiator

The conversation around AI has often centered on which models are winning. That conversation will matter less over time. Models will improve. Capabilities will expand. Access to AI will become increasingly democratized.

Governance and control will remain the differentiator. 

Organizations that establish trusted data foundations, clear oversight mechanisms, transparent decision processes, and effective human-in-the-loop controls will be able to scale AI confidently across the enterprise.

That isn't trust. It's something sturdier. And unlike trust, that’s something you can actually build.