Compressing Nemotron-3.5-SuperVL-120B to 1.4bit With Cactus Quants

We compressed Nemotron-3.5-SuperVL-120B with Cactus Quants to 1.418 effective bits per weight. The complete 124.45B-parameter checkpoint occupies 22.06 GB, including its language backbone, vision tower, projector and multi-token prediction head.

Cactus Quants is the quantization method behind Whistle, Needle and Mirage. This release brings CQ to NVIDIA's large hybrid model, combining compact vector codes with a mixed precision allocation and kernels that consume the packed projections.

Across 18 reported benchmarks, the unweighted mean relative score reduction is 3.6%. Long agentic coding is a substantial exception. DeepSWE falls from 24 solved tasks to seven and is reported separately below.

Nemotron-3.5-SuperVL-120B · Cactus Quants

22.06

GB · complete file

1.418

effective bits / weight

tokens / second

21.0

GiB · peak process RAM

36GB M5 Max MacBook Pro. 512 prompt tokens, 128 output tokens. Median warmed batch-one decode across three runs, with verified MTP. Speed rounded from 46.7 tokens/s.

The checkpoint and inference source are available together. Cactus MLX runs the model on Apple silicon through the existing cactus-compute package. On a 36GB M5 Max MacBook Pro, the compressed model reaches approximately 47 tokens/s under the measured workload above.

Cactus Quants on Nemotron

CQ stores compact codes for weight vectors together with the information needed to reconstruct their scale. In this checkpoint, projection rows are divided into groups of 128 weights. Reconstruction uses fixed lattice codebooks and a 128-point Hadamard transform, with group norms and row and column scales preserving magnitude.

The codes describe vectors jointly. A 1.5-bit allocation therefore does not require a fractional bit attached to an individual number. The corresponding packed representation uses 24 bits for each 16-value vector, or 24 bytes of codes for a 128-weight group. A BF16 group alone occupies 256 bytes. The norms and scales add storage, which is why we also report the complete-file rate.

Fig. 1

128 weights, stored as codes

BF16 group

256 bytes

CQ codes

24 bytes

The 1.5-bit setting stores 24 bits per 16-value vector. Fractional bits per weight come from encoding vectors jointly.

Figure 1. A storage schematic for one 128-weight group. Each left-hand mark represents one BF16 weight, or two bytes; each right-hand mark represents one code byte. Norms, scales and shared codebooks are additional and are included in the complete-file rate of 1.418 bits per weight.

The Hadamard transform is orthogonal and symmetric. If $q$ denotes a decoded vector in the transformed basis, the identity

\begin{aligned} \langle qH, x\rangle &= \langle q, xH\rangle, \\ H^{\mathsf T}H&=I. \end{aligned}

allows a projection to move the transform onto the input instead of first materializing every weight in its original basis. The runtime also applies the stored norms and scales. This algebra is what connects the packed representation to an efficient projection kernel; it does not remove the approximation introduced by quantization.

Nemotron does not use one precision everywhere. Routed experts account for most of its parameters, so their up and down projections use 1.5-bit and 1-bit codes. Mamba input projections and the vocabulary output head use 2-bit codes. Attention, routers, Mamba output projections, embeddings and vision linears retain 4-bit codes. Small remaining tensors keep BF16 or FP32 storage. The allocation below combines all components, including MTP.

What 1.418 bits includes

A code precision and a complete model's storage rate measure different things. This checkpoint uses mixed precisions. Its final effective rate includes codes, group norms, scales, codebooks, unquantized tensors and the safetensors header, across the backbone, vision components and MTP head.

\begin{aligned} b_{\mathrm{effective}} &=\frac{8\times\text{file bytes}}{\text{model parameters}} \\ &=\frac{8\times22{,}058{,}387{,}180}{124{,}451{,}429{,}888} \approx 1.418. \end{aligned}

Shared embeddings and the output head are counted once. File sizes below use decimal units. Runtime memory uses GiB and measures a different quantity.

Code precision	Main tensors	Parameters	Stored size including norms and scales
1-bit	Routed-expert down projections and 1-bit shared experts	59.498B	7.92 GB
1.5-bit	Routed-expert up projections	57.781B	11.34 GB
2-bit	Mamba input projections and output vocabulary head	3.578B	923.43 MB
4-bit	Attention, latent projections, routers, embeddings, Mamba output, vision/projector linears, selected shared experts and MTP input projection	3.569B	1.81 GB
BF16 / FP32	Remaining tensors, including norms and correction biases	25.305M	50.65 MB
Storage overhead	Fixed codebooks and file metadata	n/a	3.92 MB
Complete checkpoint	All components	124.451B	22.06 GB

Running the compressed projections

The checkpoint stores CQ projections in their packed representation. Our custom Metal operations use those codes with their codebooks and scales. Native MLX attention handles attention computation, while the CQ loader handles this model's packed tensors. This is a custom checkpoint format rather than a stock MLX-LM quantization preset.

Fig. 2

From packed weights to a token

The checkpoint retains compact CQ codes alongside the tensors needed to interpret them.

Figure 2. A schematic of the CQ projection path. Signal timing and matrix sizes are illustrative; they are not a latency trace or a literal kernel schedule.

Verified MTP is enabled by default for greedy generation. The MTP head proposes tokens, and the target model checks them. The reported decode measurement includes this verification work. Acceptance rate and draft overhead determine whether speculation saves time; the presence of an MTP head alone is not a throughput guarantee. Add --no-mtp to compare the ordinary decode path on your own workload.

The measured memory footprint is specific to the text workload above. Longer contexts and image inputs can change memory use and latency. The Mac result does not establish that this checkpoint fits on an iPhone.

Get started with Cactus MLX

Use an Apple silicon Mac with macOS 26 or newer and Python 3.12. Install the MLX extra from the existing Cactus package.

pip install -U 'cactus-compute[mlx]'

Generate text

Pass the Hugging Face repository name directly. The first run downloads the model; subsequent runs reuse the local cache.

cactus-mlx generate \
  --model Cactus-Compute/nemotron-3.5-120b-cq1.4bit \
  --prompt 'Write a Python function is_prime(n).' \
  --max-tokens 256

Ask about an image

cactus-mlx generate \
  --model Cactus-Compute/nemotron-3.5-120b-cq1.4bit \
  --image photo.jpg \
  --prompt 'Describe this image.'

Repeat --image for multiple images. Vision uses the tower and projector included in the same checkpoint.

Serve a local API

cactus-mlx server \
  --model Cactus-Compute/nemotron-3.5-120b-cq1.4bit \
  --port 8080

The server exposes an OpenAI-compatible chat API. It also accepts image content through image_url parts, including HTTP(S) and base64 data URLs.

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Cactus-Compute/nemotron-3.5-120b-cq1.4bit",
    "messages": [{"role": "user", "content": "Explain a binary search."}],
    "max_tokens": 128,
    "temperature": 0
  }'

Evaluation

The BF16 original and CQ model were evaluated with the same harness and settings, both served by vLLM. The Mac measurements above assess deployment performance; they are not a rerun of this benchmark suite under MLX.

Fig. 3

Eighteen benchmarks, shown individually

BF16 / CQScore, % · 0 to 100

MMLU-Pro77.50 / 71.79

MMLU-ProX Lite76.04 / 76.28

GPQA Diamond89.20 / 85.54

GSM8K94.00 / 91.67

IFEval83.33 / 82.67

IFBench71.58 / 65.31

HumanEval95.12 / 95.12

LiveCodeBench v686.07 / 81.04

BigCodeBench · thinking / complete65.26 / 61.84

BigCodeBench · thinking / instruct52.28 / 51.05

BigCodeBench · leaderboard / complete61.58 / 59.74

BigCodeBench · leaderboard / instruct52.72 / 50.18

SciCode45.83 / 41.23

GraphWalks96.58 / 88.03

AA-LCR v1.167.35 / 70.45

V*86.32 / 82.85

MathVision81.28 / 76.22

SWE-bench Verified61.40 / 63.60

Figure 3. Thin upper bars show BF16; lower bars show CQ. Scores share a 0–100% scale. These are point estimates from the recorded evaluations, not confidence intervals. DeepSWE is discussed separately below.

The 3.6% summary is an unweighted average of relative score changes across these 18 benchmark rows. It is not a 3.6 percentage-point accuracy drop or a claim about every task.

\begin{aligned} \overline{\Delta}_{\mathrm{rel}} &=\frac{100}{18}\sum_{j=1}^{18} \left(\frac{S_{\mathrm{CQ},j}}{S_{\mathrm{BF16},j}}-1\right) \\ &\approx -3.6\%. \end{aligned}

SWE-bench Verified uses mini-swe-agent with no network. BF16 solves 307/500 tasks, 61.4%, and CQ solves 318/500, 63.6%. These are observed scores, not evidence that compression generally improves agent reliability.

The model card gives exact scores, task counts and the complete evaluation qualifications. Several matter when comparing these numbers with other reports. GSM8K uses flexible extraction. MMLU-ProX uses the Lite set in ten languages and strict language-specific answer patterns. GraphWalks retains prompts that fit a 262K context with a 100K output cap. SciCode has two prefilled setup steps missing class declarations, affecting both models. BigCodeBench's leaderboard-labelled rows use a chat path rather than the official completions setup. V* and MathVision use their documented image prompts and scorers at temperature 0.6 and top_p 0.95.

Long agentic coding remains a weakness

DeepSWE with OpenCode and no network falls from 24 solved tasks for BF16 to seven for CQ. The model card reports 113 tasks, or 21.2% versus 6.2%; a saved CQ aggregate records 110 rather than 113. The three-task denominator discrepancy remains unresolved, and neither the benchmark average nor the chart includes DeepSWE.

This is a substantial observed loss on long engineering runs. A single agent run per configuration also leaves run-to-run variance unmeasured. We investigated possible mechanisms with smaller, controlled experiments, including matched repair prompts and activation interventions. The companion article, Why quantization can break multi-step workflows, separates the effects those experiments demonstrate from what remains unknown about this release.

Weights, code and citation

The Hugging Face repository contains the model, detailed evaluations and inference source. Cactus MLX is distributed on PyPI as the mlx extra of cactus-compute. The command remains cactus-mlx so the same interface can accept a repository name or local model path.

Cactus code and contributions are Apache 2.0. Upstream model materials retain OpenMDW 1.1; the release includes both license texts. For the quantization method, see the Cactus Quants paper. The model card provides the release citation.

Compressing Nemotron-3.5-SuperVL-120B to 1.4bit With Cactus Quants

128 weights, stored as codes

From packed weights to a token

Eighteen benchmarks, shown individually

Cite this post