Why quantization can break multi-step workflows

A model can solve almost every individual task and still complete surprisingly few workflows. In our short coding experiments, a 4-bit Qwen model passed 97.3% of tasks but only 78.1% of eight-task sequences. That difference is mostly ordinary compounding. Eight independent chances to succeed at 97.3% give roughly 80% success overall.

At 3-bit precision, the model sometimes repeated an expression after a test had already rejected it. In the clearest case, it agreed with BF16 at 14 of 15 answer positions, but one changed decision broke a boundary condition. Replacing an activation at that decision recovered correct code.

We began these experiments after our Nemotron CQ release solved seven DeepSWE tasks against the original model's 24. The experiments below use smaller models, a different quantizer and short, inspectable tasks. They demonstrate failure mechanisms that matter for multi-step work; they do not establish the cause of the Nemotron result.

What we measured

The main model is Qwen3-4B-Instruct-2507. We compare BF16 with fresh MLX affine weight quantization at group size 64. Only transformer projections are quantized. Embeddings, the tied output head, norms, activations and KV-cache storage retain their original precision. All decoding is greedy unless specified otherwise.

We wrote 12 expression tasks involving indexing, rounding and bounds, then checked the answers with executable tests. There are 32 sequences of eight distinct task types, with shuffled order and renamed arguments. Each task permits three attempts, including test feedback. The conversation carries forward, but an earlier answer cannot change a later requirement. These are independent programming exercises inside a shared conversation, not edits to an evolving repository. Prompts stay within 2,102 tokens, and no generation reaches its output limit.

Repeated templates are the main statistical limitation. The tables report descriptive 95% intervals from resampling whole templates, abbreviated T95. With only 12 templates, these percentile intervals can be too narrow. Our primary exploratory test swaps model labels together within each template. It assumes exchangeability under the null and does not turn 12 bug families into hundreds of independent observations. B95 denotes an exact binomial interval conditional on drawing sequences from this fixed task bank.

Compounding explains the sequence result

For independent steps with success probabilities $s_1,\ldots,s_n$, completing every step has probability

If their probabilities are equal, this becomes $s^n$. This arithmetic is the appropriate starting point before attributing a workflow deficit to a new interaction between tasks.

At 4-bit, $(249/256)^8=80.1\%$, compared with 78.1% observed. At 3-bit, $(222/256)^8=32.0\%$, compared with 31.3%. There is no need for later tasks to become harder to produce these losses. BF16's perfect task score also shows that this bank is easy for the reference model. Its degenerate T95 interval is a sample ceiling, not guaranteed reliability.

An aggregate rate can conceal unequal difficulty and dependence. We therefore shuffled outcomes within templates while preserving which templates appeared in each sequence. The 3-bit null distribution had mean sequence success 26.8%, a 95% reference range of 18.8–37.5%, and two-sided deviation $p=0.523$. The observed 31.3% is unexceptional under that shuffle.

At 4-bit, seven failures can occupy only five, six or seven sequences. The only possible success rates are 84.375%, 81.25% and 78.125%. All 18,574,248 placements are at least as far from the null mean of 79.13% as the observed result, giving $p=1$. This discrete test cannot detect modest additional amplification at 4-bit. Neither test proves independence; neither supplies evidence of extra amplification in these sequences.

A repository agent has a different feedback structure. A bad edit can alter subsequent test output, influence the next plan and change what the agent sees. We did not simulate those dependencies.

A retry is not a fresh chance

The rolling-history 3-bit model repeated an already-failed expression on consecutive attempts in 29 of 256 tasks, or 11.3% [T95 3.5, 20.7]. The corresponding counts were two for 4-bit, 0.8% [0.0, 1.9], and zero for BF16. Among each model's initially failed tasks, those counts are 29/82, 2/36 and 0/14. The models failed different subsets, so those conditional rates are descriptive rather than a matched comparison.

If each retry succeeded independently with probability $r$, three attempts would succeed with probability $1-(1-r)^3$. Repeating the same rejected expression violates the useful premise of that calculation. This dependence is within a task. It is separate from the ordinary compounding across eight tasks above.

To compare repair on identical evidence, we supplied each model with the same buggy expression and the same failing tests. We wrote one mutation per template, then created eight argument-renaming variants, giving 96 entries and 67 distinct message contexts. The mutations were written after the pilot rather than copied from 3-bit outputs, although the window-boundary mutation structurally matches the pilot error. This is not an independently designed corpus of unfamiliar bugs.

One item needs to be excluded when interpreting repair quality. The batches prompt permits zero items but never explicitly states that they require zero batches. Its feedback shows only one item, with batch sizes four and eight. Seven BF16 answers special-case zero items to one batch; all quantized configurations return the scorer's expected zero. An independent oracle agrees with the scorer over 16,448 integer pairs, but an oracle encoding the same convention cannot resolve prompt ambiguity. We remove this template from every repair configuration and comparison below. The figure can restore it for inspection.

The paired comparisons below use the 88-entry cohort. Wins and losses count cases where only the first or second configuration succeeds. Differences are percentage points. McNemar treats entries as independent pairs; the template-swap test preserves their grouping.

Removing the ambiguous item strengthens the 3-bit deficit and reveals a smaller, uncertain 4-bit deficit. The eight 4-bit losses occur in just three templates. Including batches had offset them with seven wins. Full MLP restoration no longer beats BF16 once that item is removed.

The same exclusion gives 107/110 correct BF16 sampled repairs, 97.3% [T95 92.7, 100.0], against 75/110 for 3-bit, 68.2% [40.9, 90.9]. These are five draws at temperature 0.6 and top_p 0.95 from each of 22 entries, selected as the first two renaming sets across the remaining 11 templates. The difference is −29.1 points [−56.4, −4.5], with nominal paired $p=1.94\times10^{-8}$ and template-swap $p=0.125$. Repeated draws from the same contexts do not provide 110 independent bugs.

How much does restoration location matter?

Restoring all MLPs returns 2,689,597,440 parameters to BF16, about 74% of the projections. Restoring attention returns 943,718,400, about 26%. Their raw difference cannot establish that MLPs matter more per restored parameter.

We also tested three depth-spanning MLP selections, each containing 946,339,840 parameters, only 0.28% more than the attention budget. Selection $k$ restores all MLP projections in layers whose index modulo three equals $k$, plus the gate and up projections in one additional layer numbered $(k+1)\bmod3$. The selections were fixed before their results were measured.

The second selection gains 15 repairs over the first with no losses, +17.0 points [T95 0.0, 39.8]. Its nominal McNemar $p=0.000061$ becomes 0.000488 after Holm adjustment across all nine repair comparisons. But those wins come from only three templates, and the template-swap $p=0.25$. None of the nine template-swap comparisons survives Holm adjustment. Placement sensitivity is visible on this bank; a benefit across task families is not established. The full analysis includes all nine comparisons, both cohorts and per-template counts.

Following one wrong token

The simplest pilot failure asks for the exclusive end of a window, capped at a total length. With start 8, size 5 and total 10, the answer must be 10. BF16 generates min(start + size, total); 3-bit generates min(start + size, total + 1) and returns 11.

Of 32 pilot first attempts, 3-bit failed nine. Eight had correct BF16 answers, comprising four retry cases, three alignment cases and this window case. We selected the window case during exploratory work; it was first in seed and task order, but the selection rule was not registered beforehand.

Under identical answer prefixes, the models agree on the most likely token at 14 of 15 positions. At the decisive position, BF16 puts 99.68% probability on ), while 3-bit puts 99.92% on +. This is a confident change of choice. Forcing the closing parenthesis once and then continuing normal 3-bit generation produces correct code.

Unpacking the same 3-bit weights and using dense BF16 linear operations reproduces the error. A discrepancy in the packed MLX kernel therefore cannot explain it by itself. This control concerns MLX affine quantization, not the separate CQ runtime.

We replaced one 3-bit activation with its BF16 counterpart at the input position responsible for the next-token decision. Each intervention is a separate forward pass. We swept all 36 layers for attention outputs, MLP outputs and the residual stream after the complete block. The cached prefix remains 3-bit.

Attention replacements rescue the next decision at layers 23 and 24; MLP replacements do so at 21, 23, 32 and 35. Layer-24 attention gives the correct token 49.81%, enough to win by plurality rather than majority.

Post-block residual replacements fail at every layer from 0 through 20. Later quantized computation can rebuild the wrong choice despite a BF16 current-token state. The transition is not monotonic.

Layer 22 is a clear reversal, not a near-miss concealed by a binary score. A layer-23 replacement still leaves 12 quantized blocks, layers 24–35, and they preserve the correct choice. Only replacements near the output mainly function as positive controls. At layer 35, the correct-token probability exactly equals the original BF16 value because only the unquantized final norm and head remain. This is a built-in check of the patching harness.

These results suggest a sensitive transition around layers 21–23 without identifying a single defective block. The imported state contains upstream BF16 effects, and the prefix cache remains quantized.

Layer-23 attention and layer-21 MLP replacements also recover correct completed code. Testing each at the last eight prompt positions and all 13 shared answer-prefix positions finds success only at the decision-input position. All 20 earlier positions fail for each component. A sham replacement with the original 3-bit activation leaves logits unchanged and the answer wrong.

Pattern and values have to work together here

Attention can be written as

At layer 23, we restore the BF16 pattern $A$, the BF16 values $V$, or both, while leaving the output projection quantized. This tests whether either imported component alone is sufficient at this decision.

Neither component alone changes the winner. Restoring both does. The manual 3-bit attention reconstruction has relative output RMSE 0.000616 against native attention and preserves its wrong top token. The intervention demonstrates joint sufficiency of the imported pattern and values in this case. It cannot assign the error solely to local query/key or value weights because both imports carry upstream effects.

The clearest example is not the typical failure

We checked every fresh-history first-attempt failure with a correct matched BF16 answer. The fresh run has 85 initial 3-bit failures, distinct from the rolling run's 82. BF16's fresh first answers are correct on 64 of those 85 tasks. We compare next-token choices along those correct BF16 answers; the remaining 21 lack a correct reference and are excluded.

Only nine of 64 cases have a single disagreement, and all nine are window-boundary tasks. That is 14.1%, with nominal binomial 95% interval 6.6–25.0%; resampling the five represented templates gives a much wider 0.0–50.9%. BF16 replay matches every recorded reference token. The selected case makes an intervention easy to interpret, but most eligible failures contain several disagreements. Disagreements along a supplied reference answer are not a count of causal errors in a freely generated completion.

Counterexamples constrain the explanation

Keeping history improves final task counts for all three Qwen configurations. BF16 rises from 253/256 to 256/256, 4-bit from 237/256 to 249/256, and 3-bit from 221/256 to 222/256. The respective paired changes are +1.2 points [T95 0.0, 3.6], +4.7 [−3.1, 13.3] and +0.4 [−11.5, 13.1]. Short conversation history does not explain the failures here.

A small Nemotron-4B pilot also cuts against a universal precision ranking. Over four sequences, BF16 solves 29/32 tasks and completes 2/4 sequences. The 4-bit model solves 32/32 and completes all four; 3-bit solves 23/32 and completes none. The 4-bit improvement has three paired wins and no losses, nominal $p=0.25$ and template-swap $p=0.5$. It is not an established benefit, but the closest architecture we tested does not reproduce the Qwen 4-bit ordering. Full intervals are in the accompanying analysis.

Even within Qwen, 4-bit fails five rolling-history backoff tasks that 3-bit passes. On repair, it solves ceiling division in 5/8 cases against 3-bit's 8/8. Precision effects depend on the task and model, not just the number of bits.

The bit rates also need comparable accounting. MLX's groups store a 16-bit scale and a 16-bit bias alongside the codes. With 64 weights per group,

Our 3-bit and 4-bit projections therefore occupy about 3.5 and 4.5 effective bits per weight. CQ's language-backbone projections occupy 1.3814 effective bits per weight, including codes, norms, scales and shared codebooks. The complete Nemotron checkpoint is 1.418 bits per parameter. The Cactus Quants paper describes the method used for that release. Equal accounting does not make different architectures, quantizers or precision allocations interchangeable.

What this says about DeepSWE

The model card reports 24/113 BF16 successes, 21.2%, and 7/113 CQ successes, 6.2%. A saved aggregate instead records CQ as 7/110. The original paired task records were unavailable for this analysis, so we cannot resolve those three tasks or measure the overlap of successes.

Conditional on 113 common independent task pairs, the gap is 15.0 percentage points. Across all possible overlaps of the 24 and seven successes, exact McNemar $p$ ranges from 0.0000153 to 0.00333. A conservative 95% envelope for the paired gap is 2.3–31.6 points. Even counting all three missing CQ tasks as successes gives a 12.4-point gap and a maximum paired $p$ of 0.0243 under those assumptions.

That calculation addresses task sampling conditional on the observed runs. Each configuration appears to have only one agent run. It does not measure agent run-to-run stochasticity, and it cannot replace a second BF16 run or reconciliation of the task records. No full DeepSWE evaluation was run for this article.

The mechanisms we can demonstrate are narrower. Small per-task deficits compound across required steps. Some failed repairs repeat rather than provide independent chances. In at least one inspected case, quantized weights change a decisive computation despite high agreement elsewhere, and targeted activation replacements recover the correct answer. These effects occur in short contexts, so long context is not necessary for them. Long context may still contribute at DeepSWE lengths.

Related work

Dong and colleagues' ACBench evaluates compressed models across workflow generation, tool use, long-context understanding and real-world applications. Its GPTQ/AWQ results distinguish relatively preserved workflow and tool-use scores from larger real-world application losses. This supports evaluating complete agent tasks separately from short-answer benchmarks; it does not identify the mechanism behind our release's DeepSWE result.

Afrin, Haque and Mastropaolo's study of quantization for code generation compares six methods on Qwen2.5-Coder and CodeLlama. It finds substantial variation by method and prompt complexity, including configurations that match or exceed the reference. PrismML's Ternary Bonsai release limitations separately caution about long-horizon coding. A release limitation is useful deployment context, not a controlled causal experiment.

Quantization methods make different choices about which errors to reduce. GPTQ uses approximate second-order information for weight reconstruction; AWQ uses activation-aware scaling; QuIP# combines Hadamard-based incoherence processing with lattice codebooks. Our controlled small-model study uses MLX affine quantization, not those methods or CQ. Its results should not be read as a ranking of quantizers.

The activation interventions follow the general logic of causal tracing, exemplified by Meng and colleagues' work on factual associations in GPT. Here the source and recipient runs differ in weight precision rather than a corrupted factual prompt. A successful replacement identifies a state sufficient to alter the chosen decision under that intervention; it does not by itself localize the responsible weight errors.

Evidence and statistical details

The complete plain-text analysis contains the per-template results, all paired restoration comparisons, both repair cohorts, the full layer-probability table and interval construction. The figure data provide the numerical values behind the interactive charts. Bootstrap intervals use 50,000 template resamples with seed 20261112; label-swap tests enumerate all assignments across the included templates. All comparisons remain exploratory.