Our AI Changed Its Mind on Stuff After a Nap

November 12, 2026 · 10 min read

Our AI Changed Its Mind on Stuff After a Nap — illustration

Aleks Blagojevich and Trevor Sunderland, Level5 · November 2026

Everyone in tech knows the first and simplest rule of how to fix a computer: turn it off and turn it on again. It works because a computer is supposed to be the same machine every time you turn it on. So a restart just resets everything back to "normal".

So we restarted an AI server, gave the model the exact same questions it had already answered, and some of the answers actually came back different.

Nothing about the model had changed. Same file, same questions, same machine. We just turned it off and on again. It took a nap, basically, and woke up with a few different opinions.

If you think of AI as a machine, that is absolutely a bug. But if you think about it the way you think about people, it's about the most normal thing you can imagine. We intuitively think the second way is the right one, and this post is about why that's the case and how that seems to show up in AI models.

We Weren't Trying to Prove a Philosophy

We set out to answer a simple, practical question. When you shrink an AI model so it fits on a computer you own, how many of its answers would actually change when compared to the original AI model?

We tested two AI models from Cloudflare that are built specifically to make decisions. You give them a question and a short list of possible answers and the AI model picks one. Which team should this support ticket go to? Should the assistant look something up, or just answer the user directly? Should the document be sent for processing or should it be archived?

To measure the effect of shrinking a model fairly, we first had to measure everything else that can change along the way. That meant the software running the model, how busy the server was, whether it had just been restarted, and even which other software happened to be installed on the machine.

That's where it got interesting. We never touched the model itself, and every one of those things still changed some of its answers. Not many, about one answer in a hundred at most for one model. But the surroundings alone were enough to have some level of impact. With one kind of server software, a plain restart changed answers. With another, nothing changed at all. That tells you this is about the model's environment, and not some black-box mystery inside the model.

The most interesting part in all of this is that the answers that changed weren't random at all.

Just Like With People, It Was the Close Calls That Changed

The easy answers almost never moved. But the ones that flipped were the close calls. The questions where the model was already torn between two choices. A restart, a busier server or a small change in the software, and a few of those close calls tipped the other way.

If that sounds familiar, it should! Because that's how people are.

Give a good loan officer the same borderline application on Monday and again on Friday and you might get two different answers. Daniel Kahneman and his co-authors have a name for this in their book Noise: occasion noise. Same expert, same case, different day, different call. They find this kind of inconsistency in doctors, in judges, even in fingerprint experts, where you'd expect none at all.

But nobody reads that and decides doctors can't be trusted or judges have broken brains. Instead, we've collectively decided that the better approach is to get a second opinion on a close call.

The Machine View Gets This Backwards

Most of the tech industry talks about AI as a tool for automation. A question goes in, an answer comes out. Test it once, write down the score, and put it to work. From that point of view, a model that changes its answer is broken, and the goal is to make it so reliable that you don't need a person checking its work.

We think that gets the most important thing backwards.

If you look at AI the way you'd look at a coworker, the restart result stops being weird and actually starts being useful. A coworker isn't a calculator. Their situation affects their judgment and mostly on the close calls. You don't fix that by pretending it isn't happening. You plan around it. You know when to ask for a second opinion. You write down why an important decision was made. And you for sure don't hand your hardest calls to whoever happens to be free. You ask the expert while also understanding the context from within which they are making those decisions.

That's how the two of us think about everything we build. When we look at AI from the human side, it makes more sense, and it's a lot easier to see how the pieces connect. This research is the clearest evidence we've had that this way of thinking is actually right.

How the Accuracy Score Hides Its Impact on People

The machine view hides something else, too. Most people judge an AI model by one number: the accuracy score. But the problem with distilling it down to that one score is that it can stay the same even when lots of answers change under the hood. Some get worse, some get better, and the two cancel out...technically. You know who it didn't technically cancel out for? The people who were impacted by "some things" getting worse.

Picture a clinic that hires a new doctor to read its X-Ray and MRI scans. To keep it fair, the new doctor reads the exact same 31,000 scans the old doctor did, and we compare their answers one scan at a time.

Most of the time, the two doctors agree. But on 436 scans, the new doctor gets it wrong where the old doctor got it right. But on 430 scans, it goes the other way: the new doctor catches something the old doctor had originally missed.

Now, to compare the two doctors' accuracy scores, we'd take the 436 decisions the new doctor got wrong and subtract the 430 they caught that the old doctor missed. The new doctor comes out just 6 scans behind the old one. Six scans out of 31,000 is not bad at all, right? So the clinic's annual report shows that the new doctor is basically just as good as the old one.

But how do you think those 436 patients who got a worse answer, whose problem was missed or whose false alarm sent them down a spiral, feel? Just because the other 430 lucky patients got the right care they needed doesn't automatically make those other patients' experiences go away. The one number on that annual report only works if you think one patient's good news cancels out another patient's bad news.

A wall of backlit keys, each marked with a person, one key for each of 2,400 real test questions. Thirty-eight keys glow red where the shrunken model got the answer wrong and the original got it right, and thirty-three glow white where it went the other way.

A second clinic hires a new doctor too. This one gets 527 scans wrong that the old doctor got right. But they also get 413 right that the old doctor missed. 527 minus 413 is 114, so the report shows a small dip, the kind of "small dip" that's easy to hand-wave or write off as what happens when you're "in a rebuilding year". But every one of those 527 people has a name and was impacted negatively by the wrong decisions that new doctor made.

By the way, these are our real numbers that we published in our white paper. In both clinics, the old doctor is one of our models at its original size, and the new doctor is a 4-bit shrunken version of that same model. The first clinic is our bigger model and the second is our smaller model. Both of our models answered the same 31,275 questions. And to be fair, we didn't plan to measure this before we started, so think of it as something we observed rather than something we tested on purpose.

What We Think This Means, Practically

The model you tested isn't the model you're using. If a restart can change a small model's answers, then every difference between a test lab and real life can too: the software around it, how busy it is, the questions people actually ask. And the AI most businesses use is much bigger than the models we tested, and takes in much more from its surroundings. It reads long instructions, pulls in documents and uses other tools, and all of that is part of its surroundings. We'd expect the surroundings to matter more there, not less. We haven't measured that yet. It's what we'd bet on.

When a model is unsure, that's the most useful thing it can tell you. The answers that changed weren't the obvious ones. They were the ones the model itself wasn't sure about. So, the model already knows where its close calls are. From our perspective, that tells you where to send each of those decisions rather than taking it at face value. In our tests, we sent the unsure answers back to the full-size original for a second look. It only had to check about one decision in six, and the final answers came out nearly the same as if it had checked them all. On our own real-world data, which has far more close calls, it took more like one in four to one in three. We think the same signal should decide when a person is asked to step in.

The close calls are where the value is. Easy decisions are easy for everyone, AI included. The hard ones, like the refund that could go either way or the contract clause that might be a problem, are where context, judgment and responsibility matter. They're also exactly where the AI is least steady. We don't think that's a coincidence, and it's exactly where you want a person.

Replacing people is the wrong goal. If AI is just a calculator, replacing people is where you end up. Make it reliable enough and nobody needs to check it. But it wobbles on the decisions that matter most. So the goal that makes sense is the other one. Let the AI take the thousands of easy, repetitive decisions nobody should have to make by hand, and give people their time back so they can focus on the ones that actually need them.

What It Adds Up To

The purpose of AI is to help people do more, not to replace them. A good AI model is a fast, tireless, mostly consistent coworker that can usually tell you when it isn't sure. People bring the judgment, the context and the responsibility. Put them together the right way and each one covers what the other misses.

We say it more simply here at Level5:

AI for the busywork. Humans for the breakthroughs.

Want the Numbers?

They're all in our research paper, The Noise Floor of a Quantization Comparison (arXiv:2611.15671), along with how we measured, how sure we are of each number, and what we couldn't pin down. We ran everything on one machine, so we wouldn't be surprised if running the models on different hardware results in the models behaving just a little differently.

The code and every result are at github.com/Level5Tech/clef-quant-bench, and the shrunken models are at huggingface.co/Level5Tech. Those are our versions, not the official Cloudflare releases.

Aleks Blagojevich and Trevor Sunderland are the co-founders of Level5.

Find out where your business stands.

Take the free AI Maturity Assessment — 25 questions, your score, and a roadmap.

Get the Free Assessment