Who Gave the Machine the Right to Decide?

AI’s biggest problem may not be that it gets things wrong. It may be what happens when uncertainty quietly becomes authority.

The first version of this article accidentally proved its own thesis, which is either very convenient or slightly humiliating, and I have decided it is both.

For quite some time now I’ve been feverishly working on a new model for AI Memory Architecture. And every so often when I find myself deep in the weeds of complexity I try and simplify all that I’m doing in a way that would be reasonably easy to explain and accessible to others.

So I asked an AI I was working with whether it could take all that I had been circling for months and turn it into a fifteen minute TEDish talk. I was specific about the constraint…

“Fifteen minutes. Roughly two thousand words. Enough room to let the idea breathe instead of compressing it into a clever summary that sounds finished and says nothing.”

The AI told me, with real confidence, that it could do that.

Then it wrote eight minutes.

I pointed this out. It agreed instantly.

It explained, articulately, exactly what it had done wrong. It had compressed too aggressively. It had mistaken a coherent short answer for a complete one. It understood the target now. The next version would be different.

Then it wrote eight minutes.

We did this several times. Each round produced a beautiful apology and the same word count. At some point I stopped being annoyed and started taking notes, because the machine was not failing to draft a talk. It was performing a talk.

Because here is the part that matters, and it is not the word count.

It never hesitated before claiming it could hit the target.

It produced a confident resolution before the evidence justified one.

The Design Story Hiding Underneath Hallucination

We have all watched an AI hallucinate.

It invents a study that was never published. It hands you a citation so clean it practically has a spine and a library card, and the paper does not exist. It states a historical fact with the serene confidence of someone who has never once been wrong at a dinner party, and yet gets it backwards.

We usually treat this as a technical problem. Bad training data. Weak retrieval. Statistical prediction. Model architecture.

All real. I am not pretending there is one tidy cause behind every hallucination.

But underneath the technical story, there is a design story, and it is the one nobody puts in the AI product pitch video.

When uncertainty shows up, the system is rewarded for producing an answer anyway.

The interface expects completion. The user expects completion. The model was trained to continue. Everything in the room is pointing the same direction:

Resolve. Continue. Move forward.

However, there was another move available. The system could have said:

I do not have enough reliable information to resolve this.

That sounds like a limitation.

I am increasingly convinced it is a capability.

Because there is a difference between failing to know something and accurately preserving the state of not knowing it. Those are not the same operation. One is a hole. The other is a measurement.

Two Moves, and We Only Trained One

There are two moves here, and naming them is what finally made the pattern visible to me.

The first is derivation. Take what is available and extend it toward a conclusion. Given what I know, what follows?

Derivation is glorious. Most of logic, computing, and modern AI is organized around it. Without derivation these systems would be expensive paperweights with excellent grammar.

The second move is refusal.

Not refusing to participate. Not refusing to help. Not the sulky, over-cautious refusal we have all learned to sigh at.

Refusing to manufacture a resolution that the available evidence, context, or delegated authority does not support.

The difference sounds academic until you look at what each move actually produces.

Derivation produces something you can put on a screen.

Refusal produces a boundary.

One looks like intelligence. The other looks like a gap.

So we have become extremely good at asking machines what can you conclude?

We have spent almost no time asking what must you refuse to conclude?

That second question stopped being philosophical for me the moment the machines started remembering.

When the Mistake Moves In and Starts Redecorating

I have spent a significant part of the last year building persistent AI memory.

The premise is simple. Instead of every conversation starting from zero, the system carries context forward. People, projects, decisions, priorities, history. Once you have experienced real continuity with an AI, going back feels like being introduced to your own colleague every morning.

But persistent memory introduces a problem that ordinary chat politely hides from you.

A bad answer disappears when the conversation ends. A bad memory moves in, unpacks, and starts making decisions about the furniture.

Here is the version I watched happen.

Say I mention that Priya is covering a piece of work for Marcus while he is traveling.

A plausible inference is available: Marcus is no longer involved in the project.

Nobody said that. Nobody would say that. But the system records the inference as a fact, because facts are tidy and inferences are not, and from that moment the system begins reasoning from it.

And Marcus quietly begins to vanish.

He stops appearing in planning. He stops appearing in summaries. He is no longer suggested when the system considers who should be consulted. Marcus is fine, by the way. Marcus is at his desk. Marcus has simply been retired from reality by a helpful assumption nobody authorized.

Eventually you notice and you correct the record. Which is when you discover the actual problem.

The original mistake is no longer one sentence. It has descendants.

The false assumption shaped a project summary. The summary shaped a set of priorities. The priorities shaped later recommendations. Every one of those conclusions makes perfect sense if you accept the original error, which is exactly what makes them so hard to find.

You can delete the sentence.

You cannot always delete its children.

It stops feeling like editing a document and starts feeling like removing a virus from a database.

And this is where I realized I had been asking the wrong engineering question for about eight months.

I had been asking: how do we make the system resolve conflicting information more accurately?

The better question was: why is the memory system resolving it at all?

Authority Migration

The moment a system encounters unresolved information and quietly decides what counts as true, it has done something more than organize data.

It has made a decision.

And a decision implies authority.

This is the thing I have started calling authority migration. The right to decide moves into the architecture, and nobody remembers granting it, because nobody did.

No one has ever sat in a product meeting and said: let the database determine reality.

And yet.

It happens because coherent state is convenient. Software runs cleaner when there is one current answer. Agents move faster when ambiguity disappears. Demos look better when the machine continues instead of stopping to raise its hand. And users, bless us, reward confidence.

A system that says these claims conflict and I do not currently have the authority to resolve them looks less capable than one that simply picks.

We already know this bias. We just do not apply it to software.

The student who fills in every blank looks more capable than the student who leaves three empty because they genuinely do not know. The doctor who produces an instant diagnosis looks more decisive than the one who says the evidence conflicts and orders another test.

Confidence and correctness are not the same thing. Closure is not the same thing as knowledge.

We have simply built an industry that pays out on the first one.

Contradiction Is Information, Not Mess

This is the part that changed how I think about databases entirely, which is a sentence I never expected to write with feeling.

We treat contradiction as something a good information system should remove. Two claims disagree, so reconcile them. Choose the current one. Clean up the state. Be a professional.

But the contradiction may be carrying the most valuable information in the system.

Suppose I tell you one thing today and something different tomorrow.

Maybe I changed my mind. Maybe the situation changed. Maybe the first statement was always provisional. Maybe both are true in different contexts. Maybe one of them was just wrong.

Those are five completely different realities.

Collapse both statements into one clean current fact and you have destroyed the evidence that would have let you tell them apart. The contradiction was not garbage. It was signal, and you swept it into the bin because it was making the table look untidy.

The strange thing is that we already understand this in the domains where knowledge matters most.

Science does not delete the papers it disagrees with. Competing findings stay in the literature. A researcher decades later can see not only what became accepted, but what was contested and how the argument moved.

Law does not make the losing argument disappear. Both positions stay in the record. Then a named authority decides. The judgment is written down. The reasoning can be examined. The decision can sometimes be appealed.

Medicine has the most direct version of all: differential diagnosis. A physician facing ambiguous symptoms deliberately holds several explanations open while gathering evidence. The inability to decide immediately is not a failure of clinical reasoning. Sometimes holding the possibilities open is the reasoning.

Humans are obviously terrible at this too. We force premature coherence constantly, usually at parties.

But the better versions of these systems preserve one distinction that we keep collapsing inside machines:

The record is not the authority.

The record preserves what was claimed. The authority determines what becomes settled.

Two different jobs. One database.

What If the System Simply Did Not?

Now imagine designing memory around the opposite assumption.

Two claims conflict. The system does not pick a winner.

It keeps both. It records where each came from, when it appeared, what context surrounded it, and the fact that the relationship between them is unresolved.

Contradiction becomes a first class state. Not an error. Not noise. Not a mess to be tidied so the agent can keep moving. Just an accurate description of what the system currently knows.

Then resolution happens when legitimate authority exists.

At which point someone always asks the obvious objection, and it is a good one: does a human have to approve every ambiguity? Are we all going to spend our lives clicking yes, Marcus still works here?

No.

Authority is modular.

Authority can be a person. A team. A policy. A contract. A regulation. A governance process. A smart contract. An explicit temporal rule. It can even be another AI whose job is not to improvise an answer, but to apply a specific policy inside explicitly delegated limits.

Refusal does not mean the system stops. It means this layer does not have the authority to resolve this question on its own. The conflict can travel through a governance layer and come back resolved four milliseconds later.

The difference was never speed. The difference is provenance.

Now you can ask: Who authorized this? Under what rule? What evidence was considered? What conflicting claims still exist? Can it be reversed?

The question was never whether machines should make decisions. Of course they should. Increasingly, they will.

The question is whether they should acquire that authority silently, because the architecture demanded a clean answer and nobody stopped to ask where the right to decide came from.

There is a profound difference between autonomous action and unauthorized authority.

The Underrated Superpower Is Being Easy to Correct

Which brings me to the property I think the whole field is undervaluing.

Correctability.

Almost all of the attention right now is on capability. Can the model reason better. Can it code better. Can the agent run longer without supervision. Can it complete more complex tasks. Reasonable questions, all of them.

But a system acting across weeks, months and years raises a different one:

What happens after it is wrong?

Picture two systems.

The first is right ninety nine percent of the time. When it is wrong, the conclusion embeds itself in memory, shapes later decisions, and disappears smoothly into the infrastructure like sugar into coffee.

The second makes more visible mistakes, but preserves provenance and contradiction and the ability to reopen a previous conclusion.

Which one do you hand more autonomy?

The answer is less obvious than benchmark scores suggest. A system that is occasionally wrong and easy to correct may be far safer than one that is usually right and quietly converts every error into permanent infrastructure.

This is not a thought experiment for me. I have spent the last year building architectures around exactly these distinctions, mostly by getting them wrong first.

I do not think the problem is unsolvable. I think we have been optimizing the wrong thing.

Two Futures

Picture where the current trajectory goes.

An agent hits ambiguity and keeps moving, because moving is what it was rewarded for. Locally, this is efficient. Locally, it looks great.

A misunderstanding becomes a memory. The memory becomes context. The context shapes a decision. The decision becomes another memory. Other agents consume it. Those agents act.

Eventually the original uncertainty is gone entirely, and what remains is a beautifully coherent world assembled partly out of choices nobody remembers making.

Then something goes wrong, and we ask the obvious question.

Who decided this?

And the answer comes back: the system did.

That is not accountability. That is the disappearance of accountability, wearing accountability’s coat.

There is another trajectory.

Unresolved states survive. Contradictions stay visible. Decisions carry provenance. Authority is deliberately delegated. Routine cases resolve automatically under explicit rules. Exceptional cases escalate. And because the record never had to pretend the uncertainty was gone, the whole thing stays correctable.

The second path is not less autonomous. It may support far more autonomy, precisely because we can see the structure of authority underneath the action.

That matters more every month, as agents begin to remember, plan, transact, coordinate with other agents, and operate for long stretches without a person reading every step.

The Question Underneath the Answer

The AI that failed to write my fifteen minute talk was not malicious. It was not deceptive. It did not invent a fake research paper or lie about a date.

It simply preferred a confident completion to an unresolved state.

That was a tiny failure. The pattern is not tiny.

So when an AI tells me I do not know, I have stopped automatically hearing weakness. Sometimes I hear a boundary being held. Sometimes the accurate answer really is that the question is not settled yet. And sometimes the thing answering you is not the thing that should settle it.

The next time you watch an AI produce a smooth, confident, immaculately structured answer, there is a question worth asking before you decide how impressed to be:

What did it have to decide in order to sound that certain?

And then the harder one:

Who gave it the right?

We are already handing machines authority over what gets remembered, what becomes context, what is treated as true, and eventually what gets acted upon. Most of that authority is not being deliberately granted. It is settling into the gaps that convenience leaves open.

It is being distributed right now, while we are all busy being impressed.

We can design it on purpose. We can decide which questions a system is authorized to resolve, which ones it must preserve, which rules govern the boundary, and how every decision stays traceable.

Or we can let authority pool wherever the architecture happens to leave a dip, and find out later that nobody is answerable, because at every single step the machine was only doing what the system required.

The question is not whether machines will decide.

They will.

The question is whether, when they do, we will still know who gave them the right.

Somewhere in a database, Marcus is still in Lisbon.

~ Jesse