Defeating Nondeterminism in LLM Inference

Setting temperature to 0 does not in practice make LLM API outputs deterministic.

“LLM APIs are still not deterministic in practice”

assertion · unclear

consistent · high confidence — This is a widely documented phenomenon across OpenAI, Anthropic, and OSS serving stacks (vLLM/SGLang issue trackers, community threads) even at temperature 0 / greedy decoding. Matches my own knowledge of repeated reports of this behavior predating this post.

To check: Run identical temperature-0 requests against a commercial API or self-hosted vLLM/SGLang server multiple times and diff outputs.

The widely repeated 'concurrency + floating point' explanation for LLM inference nondeterminism is incomplete.

“While this hypothesis is not entirely wrong, it doesn’t reveal the full picture.”

contrarian · unclear

consistent · medium confidence — The 'concurrency+floating point' explanation is popular but imprecise; it conflates a mechanism that can cause nondeterminism (atomics) with one that usually doesn't apply to the LLM forward pass. This corrective framing is a reasonable and, at the time, underappreciated point in ML systems circles.

To check: Survey GPU kernel implementations (cuBLAS/cuDNN/Triton) for atomic-add usage in standard forward-pass ops.

Repeating the same matmul on the same GPU data yields bitwise identical results, so concurrency plus floating point alone does not produce nondeterminism.

“even on a GPU, running the same matrix multiplication on the same data repeatedly will always provide bitwise equal results”

assertion · unclear

consistent · high confidence — For a fixed input shape/dtype/hardware/software version, cuBLAS and similar libraries select a deterministic algorithm and execute it identically each time absent atomics; this matches known GPU numerics behavior.

To check: Reproduce the given PyTorch snippet (1000-iteration bfloat16 matmul assertion) on any CUDA GPU.

LLM forward passes typically contain no atomic adds, because batch-dimension parallelism and split/semaphore reductions suffice.

“in the typical forward pass of an LLM, there is usually not a single atomic add present.”

assertion · unclear

plausible · medium confidence — Consistent with standard high-performance kernel design (data-parallel batching, split/tree reductions with semaphores instead of atomics) as commonly used in cutlass/Triton kernels, but I cannot independently verify 'usually not a single' across all inference stacks and kernel libraries.

To check: Audit kernel source for vLLM/SGLang/TensorRT-LLM forward-pass ops (matmul, RMSNorm, attention) for atomicAdd usage.

Atomic adds and concurrent finish order play no role in the nondeterminism users observe from LLM inference.

“concurrency (and atomic adds) end up being completely uninvolved in LLM inference nondeterminism!”

contrarian · unclear

plausible · medium confidence · novel — This is the article's central corrective claim and follows logically from claims 2-3, but 'completely uninvolved' is a strong universal statement that depends on the absence of atomics holding across all deployed inference kernels, which I can't fully verify though it aligns with known kernel design patterns.

To check: Comprehensive kernel audit across major inference engines confirming zero atomic-add usage in forward-pass ops.

Given identical inputs, an LLM forward pass produces identical outputs across runs.

“Thus, the forward pass in an LLM is in fact “run-to-run deterministic.””

assertion · unclear

consistent · medium confidence — Given fixed algorithm selection and no atomics, this is the standard expectation for deterministic GPU kernels; matches known cuBLAS/cuDNN behavior for fixed shapes, though algorithm auto-tuning/heuristics could occasionally introduce exceptions not addressed here.

To check: Repeat forward passes with identical batched inputs on a fixed vLLM build and diff logits.

Widely used Triton FlashAttention backward kernels avoid atomics at a cost of 40% extra FLOPs, diverging algorithmically from the FlashAttention-2 paper.

“The standard Triton implementation does additional recomputation in the backward pass, avoiding atomics but costing 40% more FLOPs!”

quantity · unclear

plausible · low confidence — It is known that some Triton FlashAttention backward implementations recompute intermediate values to avoid atomic accumulation into dK/dV, differing from Tri Dao's CUDA implementation; the general mechanism is credible but I cannot independently confirm the precise 40% FLOP figure.

To check: Compare FLOP counts of Triton FlashAttention-2 backward kernel vs. the algorithm in the FlashAttention-2 paper.

A request's output depends on concurrent requests because the forward pass is not batch-invariant, not because information leaks across batches.

“it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass.”

assertion · unclear

consistent · high confidence · novel — The empirical demo (matmul row-vs-batch discrepancy of ~1669) is a real, reproducible phenomenon caused by shape-dependent kernel/algorithm selection (e.g., split-K, tile size, tensor-core instruction choice); framing this as 'batch invariance' rather than 'information leakage' is an accurate and clarifying distinction.

To check: Reproduce the linspace matmul batch-size comparison snippet on a CUDA GPU.

Varying server load, which changes kernel batch size, is the main cause of nondeterministic LLM inference endpoints.

“the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies!”

assertion · unclear

plausible · medium confidence · novel — This is the article's headline thesis; it's a coherent synthesis of established facts (batch-size affects kernel numerics; load is unpredictable) but the claim that this is the 'primary' cause versus other lesser factors (algorithm/version drift, hardware nondeterminism in edge kernels) is asserted rather than exhaustively ruled out.

To check: Run identical single requests against a live inference endpoint under varying concurrent load and check output divergence versus a load-invariant deterministic build.

Load-driven batch-size nondeterminism affects CPU and TPU inference endpoints too, not just GPUs.

“This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism.”

assertion · unclear

plausible · low confidence — The underlying mechanism (shape-dependent reduction strategies breaking batch invariance) is not GPU-specific in principle, but the article offers no CPU/TPU measurement to back this generalization, so it's an unverified extrapolation.

To check: Run the same batch-invariance matmul test on TPU (via JAX/XLA) or CPU (via oneDNN/MKL) kernels and check for batch-size-dependent numerics.

Making a transformer batch-invariant requires fixing only three reduction operations, since pointwise operations can be assumed batch-invariant.

“we only need to worry about the 3 operations that involve reductions — RMSNorm, matrix multiplication, and attention”

assertion · unclear

consistent · high confidence — This matches standard transformer architecture: pointwise ops (activations, elementwise scaling) don't reduce across elements, while RMSNorm, matmul (linear/QKV/MLP projections), and attention softmax involve reductions — a correct enumeration of where floating-point order-dependence can enter.

To check: Enumerate operator types in a standard transformer forward pass (e.g., Llama/Qwen architecture) and check which require cross-element reduction.

In-switch NVLink-Sharp reductions behave deterministically on Blackwell and on Hopper with CUDA 12.8 or later.

“NVLink-Sharp in-switch reductions are deterministic on Blackwell as well as Hopper with CUDA 12.8+”

assertion · unclear

unverifiable · low confidence — This is a narrow, version-specific NVIDIA hardware/software fact sourced to an NCCL GitHub issue comment; I don't have detailed enough knowledge of NVSwitch SHARP determinism guarantees across CUDA versions to confirm or refute it.

To check: Check NVIDIA/NCCL GitHub issue #1497 and NCCL release notes for CUDA 12.8+ determinism guarantees on Hopper/Blackwell NVLink-Sharp reductions.

A batch-invariant single-configuration matmul kernel costs roughly 20% performance versus cuBLAS.

“Despite obtaining batch invariance, we only lose about 20% performance compared to cuBLAS.”

quantity · unclear

unverifiable · medium confidence · novel — This is a firsthand benchmark number from the authors' own unoptimized Triton kernel (explicitly noted as lacking TMA); plausible in magnitude for an unoptimized custom kernel versus a highly-tuned vendor library, but not independently reproducible from the text alone.

To check: Benchmark the released thinking-machines-lab/batch_invariant_ops Triton matmul kernel against cuBLAS across matrix shapes.

Using a single compiled matmul kernel configuration across all shapes is the simplest route to batch invariance, and is affordable in LLM inference because the model dim is large.

“the easiest way to ensure batch invariance for matmuls is to compile one kernel configuration and use that for all shapes”

assertion · unclear

consistent · medium confidence — Given the article's own analysis that split-K/tile-size/tensor-core-instruction choice varies by shape and breaks batch invariance, fixing one configuration is the logically simplest fix, consistent with known kernel-engineering tradeoffs (uniform config sacrifices some perf but guarantees identical reduction order).

To check: Inspect the released batch-invariant matmul kernel implementation for single fixed configuration across shapes.

Batch-invariant attention requires fixing the size of each KV split rather than the number of splits.

“to achieve batch invariance, we must adopt a “fixed split-size” strategy”

assertion · unclear

plausible · medium confidence · novel — This follows the article's own detailed reasoning about FlashDecode/Split-KV: fixing split count makes reduction order depend on total KV/query length, while fixing split size does not; this is a coherent original engineering conclusion consistent with described mechanics, though it depends on unreleased FlexAttention changes that I cannot independently inspect.

To check: Review the promised upstream FlexAttention 'fixed split-size' KV-splitting implementation once released.

Summing a small eight-element array in different orders yields 102 distinct floating-point results.

“there are 102 possible different results for summing this array depending on the order”

quantity · unclear

plausible · medium confidence — Floating-point sum order-dependence producing dozens to hundreds of distinct results for an 8-element array with mixed magnitudes (1e-10 to 1) is a well-known and plausible outcome, but I cannot verify the exact count of 102 without executing the code.

To check: Run the provided Python script (random.seed(42), 10000 shuffles) and count unique sums.

1000 temperature-0 completions from Qwen3-235B produced 80 distinct outputs, the modal one appearing 78 times.

“Surprisingly, we generate 80 unique completions, with the most common of these occuring 78 times.”

quantity · unclear

unverifiable · medium confidence — A firsthand experimental result (Qwen3-235B, 1000 temperature-0 completions) that I cannot independently reproduce, though it is qualitatively consistent with widely reported real-world LLM API nondeterminism at temperature 0.

To check: Reproduce the exact experiment: sample Qwen3-235B-A22B-Instruct-2507 1000 times at temperature 0 with the same prompt and count unique outputs.

Divergence between the 1000 completions first appeared at the 103rd token, with 992 continuing 'Queens, New York' and 8 'New York City'.

“we see that the completions are actually identical for the first 102 tokens!”

quantity · unclear

unverifiable · medium confidence — Firsthand token-level detail from the same experiment as claim 16; internally plausible (divergence appearing at a specific factual token like birth location) but not independently checkable without rerunning the exact experiment.

To check: Rerun the same 1000-completion experiment and inspect token-level divergence point and branch counts.

With batch-invariant kernels enabled, all 1000 sampled completions were bitwise identical.

“when we enable our batch-invariant kernels, all of our 1000 completions are identical”

assertion · unclear

plausible · medium confidence · novel — This is the paper's core empirical validation of its thesis; it follows logically from the batch-invariance argument and is internally consistent with the released open-source kernel library, though it is a firsthand result I cannot independently reproduce.

To check: Run the identical 1000-completion sampling test with the released thinking-machines-lab/batch_invariant_ops kernels enabled in vLLM.

Deterministic inference cost roughly 2x wall-clock (55s unoptimized, 42s with an improved attention kernel) against 26s for default vLLM on a 1000-sequence benchmark.

“Unoptimized Deterministic vLLM: 55”

quantity · unclear

unverifiable · medium confidence — Firsthand single-GPU benchmark numbers (26s default vs 55s/42s deterministic variants) that are plausible in magnitude for an early, non-optimized FlexAttention integration, but not independently verifiable from the text.

To check: Benchmark vLLM with and without the batch-invariant kernel patch on Qwen-3-8B for 1000 sequences of length 90-110 on a comparable GPU.

Numerical mismatch between training and inference stacks silently converts on-policy RL into off-policy RL.

“the different numerics between training and inference implicitly turns our on-policy RL into off-policy RL”

assertion · unclear

plausible · medium confidence — This tracks a real and actively discussed issue in the RLHF/RLVR community around 2024-2025 — mismatches between the sampling engine (e.g. vLLM) and training framework's forward pass introducing subtle policy drift that undermines strict on-policy assumptions; the article attributes this to other researchers' prior analysis rather than claiming it as original.

To check: Compare logprobs from a training framework's forward pass versus the sampling engine's forward pass on identical rollouts and measure divergence.

Deterministic inference is the prerequisite for bitwise-identical sampler and trainer numerics, which yields genuinely on-policy RL.

“deterministic inference enables us to also modify our training stack to obtain bitwise identical results between sampling and training, thus resulting in true on-policy RL”

assertion · unclear

plausible · medium confidence · novel — A novel synthesis connecting the batch-invariance fix to RL training fidelity; logically coherent given the premises, and the reported zero-KL result (claim 23) supports it, but it remains a firsthand, single-experiment result from the authors' own RLVR setup that I cannot independently verify.

To check: Reproduce the Bigmath RLVR experiment with Qwen2.5-VL-8B using bitwise-identical sampler/trainer numerics and measure KL divergence.

In the RLVR experiment, omitting importance weighting caused reward collapse mid-training while off-policy correction kept training stable.

“If we train without off-policy correction (i.e. importance weighting), our reward collapses partway through training, whereas adding an off-policy correction term allows training to proceed smoothly.”

assertion · unclear

plausible · medium confidence — The general phenomenon — that ignoring policy mismatch (no importance weighting) destabilizes RL training while correcting for it restores stability — is consistent with established off-policy RL theory (importance sampling bias correction), though the specific experimental outcome (reward collapse, Step 318 spike) is a firsthand result I can't independently verify.

To check: Rerun the Bigmath RLVR training with and without importance-weighted off-policy correction and check for reward collapse.

Bitwise-identical sampler and trainer produced exactly zero KL divergence throughout training, versus ~0.001 with importance weighting.

“when running “True On-Policy RL”, our KL-divergence stays flat at 0, indicating that there is no divergence between the training policy and sampling policy”

quantity · unclear

plausible · medium confidence · novel — This is the key quantitative payoff of the determinism work, directly following from achieving bitwise-identical sampler/trainer numerics; internally consistent with the article's own methodology but is a firsthand, unreplicated measurement.

To check: Measure KL-divergence in logprobs between sampler and trainer across training steps in a reproduction of this experiment.