August 28, 2026
Run the same prompt twice, at temperature zero, and you can still get two different answers.
Most people who have worked with these systems know this happens. The explanation usually offered is floating-point arithmetic — that floating-point addition is not associative, so when concurrent operations finish in a different order from one run to the next, the results differ. That explanation is not wrong exactly, but it is not the part that accounts for the variance you actually see.
Since last September two separate findings have landed, one about repeatability and one about accuracy. They get filed under the same heading — “AI is unreliable” — and they are not the same problem. Keeping them apart is worth the effort, because the thing you would do about one has nothing to do with the other.
The first finding: your output depended on how busy the server was
In September 2025, a team at Thinking Machines Lab published an analysis of why nearly every commercial inference endpoint is nondeterministic. Their conclusion was that the floating-point hypothesis “is not entirely wrong” but “doesn’t reveal the full picture.”
The actual mechanism is something they call batch invariance, or rather its absence. When your request arrives, it is processed alongside however many other requests happen to be in flight at that moment. The kernels that do the computation are individually deterministic — run one twice with identical inputs and you get identical output. But the forward pass “lacks batch invariance,” meaning the result for your request depends on the size of the batch it was grouped into. And batch size varies with load.
So the same prompt, submitted twice, can produce different text because the second time more people were using the service.
This is not a bug in the sense of something someone forgot to handle. It is a consequence of how inference is made efficient at scale. Their analysis describes what it would take to fix, and the fix has costs.
The second finding: guessing is what the scoring rewards
In April 2026, a team led by Adam Tauman Kalai published in Nature a paper with a title that states its thesis: evaluating large language models for accuracy incentivizes hallucinations.
Two parts of it are worth separating. The first is that next-word prediction during pretraining produces “statistical pressure towards hallucination even with idealized error-free data.” The pressure does not come from bad data. It comes from the objective. Regularities that recur — grammar, common structure — are learned reliably. Facts that appear rarely are not, and the model still has to produce something.
They put a floor under this. Within the setting they formalize — a pretrained base model, before post-training — Theorem 3 establishes that the error rate is bounded below by the fraction of training facts that appear exactly once. Birthdays are hard for the same reason capital cities are easy: one appears once, the other appears constantly.
The second part is about how we grade. Their meta-evaluation found that the vast majority of popular benchmarks use binary grading, which scores an abstention exactly the same as a wrong answer. If saying “I don’t know” earns zero and a guess earns a chance at one, a system optimizing for the score will guess. The incentive was built, and then the behavior was a surprise.
Why keeping them separate matters
One finding is about whether you get the same answer twice. The other is about whether the answer is right. These are independent properties. A system can be perfectly repeatable and reliably wrong. A system can also be accurate on average and still hand you something different on Tuesday than it did on Monday.
They also fail differently in practice. Nondeterminism is invisible if you only look once — there is nothing on the face of the output that tells you a second run would have differed. An inaccuracy at least has the property of being wrong, which means it can in principle be caught by someone who checks against a source. Variance cannot be caught that way, because both runs may be defensible.
The word “hallucination” has spread far enough that it now gets used for both, and that is the conflation I would push back on hardest. It describes the accuracy problem. It says nothing about the repeatability one.
What I take from it
The framing I keep coming back to is that neither of these is a malfunction. Nothing broke. Nondeterminism is what you get when inference is batched for efficiency and load is variable. Guessing is what you get when the scoring rewards a guess over an admission of uncertainty. Both behaviors are what the construction produces, from people who made reasonable engineering tradeoffs.
That reframing changes what kind of question this is. “How do we stop it from doing that” assumes a defect. The more useful question is what a process looks like when the tool’s variance is a property rather than an error — which is not a question about the tool at all.
You ran it once. What in your process would have told you that the second run said something different?
Sources
Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference,” September 10, 2025. thinkingmachines.ai
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, “Evaluating large language models for accuracy incentivizes hallucinations,” Nature 653(8116):1047–1051, April 22, 2026. nature.com
Ariadne Latorre is a Florida Supreme Court certified mediator and a software engineer. She mediates business and commercial, insurance and risk allocation, LegalTech and information governance, HOA and community, and court-ordered matters, in English and Spanish, in Miami and virtually worldwide.