Intelligence Ladders: One Set of Weights, Every Depth a Model

A watch, a Raspberry Pi and a flagship phone do not want the same model. They want the same behaviour at different sizes, and they want to pick the size at deploy time rather than at training time. The usual answer is a family: train several models, ship several files, fine-tune each one separately. Needle 3 has one set of weights, and every depth from 2 to 20 layers is a model that runs on the same engine. We call the mechanism an intelligence ladder. This post is the construction, the maths, and the numbers.

Which blocks make a subnetwork

A 20-layer Needle is a stack of blocks $f_0, \dots, f_{19}$ around a residual stream. A subnetwork of depth $d$ is a subset $S_d \subseteq \{0, \dots, 19\}$ of those blocks, run in their original order; everything not in $S_d$ is skipped. The question is which subset. Two things matter: the subnetworks must nest, so that one training run serves all of them, and the blocks must stay spread through the stack, so that a shallow subnetwork still sees early and late computation.

Both fall out of one rule. Start from the two endpoints, then repeatedly split the widest remaining gap at its midpoint:

$S_2 = \{0,\ L-1\}, \qquad S_{d+1} = S_d \cup \Big\{\Big\lfloor \tfrac{a+b}{2} \Big\rfloor\Big\}, \quad (a, b) = \arg\max_{\substack{a<b \text{ adjacent in } S_d}} (b - a)$

with ties broken toward the leftmost gap. Because each step only adds a block, the sets nest, $S_2 \subset S_3 \subset \cdots \subset S_{20}$, and because each step halves the largest hole, the coverage stays balanced at every depth.

Twenty blocks give a single fixed sequence, $0, 19, 9, 14, 4, 6, 11, 16, 2, 7, 12, 17, 1, 3, 5, 8, 10, 13, 15, 18$, and subnetwork $d$ is its first $d$ entries. Pick a depth below to see which blocks run.

Running a subnetwork

Skipping a block is the identity on the residual stream. With $\mathbb{1}[\cdot]$ the indicator, the forward pass at depth $d$ is

$x_{\ell+1} = x_\ell + \mathbb{1}[\ell \in S_d]\, f_\ell(x_\ell), \qquad \ell = 0, \dots, L-1,$

followed by the shared final norm and the shared tied unembedding. Three details make this work in the actual architecture rather than only on paper. Needle's residual is multi-lane, four parallel streams that each block reads from and writes to with a round-robin lane schedule; inside a subnetwork that schedule is renumbered over the blocks that remain, so consecutive surviving blocks still alternate lanes. The engram memory sites and the global-attention layers are properties of specific blocks, and they are kept exactly when their block is kept. And the KV cache is per block, so a shallower subnetwork is not only fewer FLOPs per token but a proportionally smaller cache, which is what a memory-bound device notices.

Training every depth at once

The naive way to get $L-1$ good subnetworks is $L-1$ loss terms per step, one forward pass each. That is too expensive at scale, and it is also unnecessary. Each step samples one path. With probability $0.8$ it is the full model and the step is an ordinary language-modelling step. Otherwise a depth $d$ is drawn uniformly from $\{2, \dots, L-1\}$, only the blocks in $S_d$ run, and the loss is scaled by how much of the model took part:

$\mathcal{L}_{\text{step}} = \begin{cases} \mathrm{CE}(p_L) & \text{with probability } 0.8,\\[4pt] \dfrac{d}{2L}\Big[\mathrm{CE}(p_d) + \lambda\, \mathrm{KL}\big(\operatorname{sg}[p_L]\,\|\,p_d\big)\Big], \quad d \sim \mathcal{U}\{2, \dots, L-1\} & \text{with probability } 0.2, \end{cases}$

where $p_d$ is the next-token distribution produced at depth $d$ and $\operatorname{sg}$ is a stop-gradient. The KL term is self-distillation: on paired steps the full model's own predictions, frozen for that step, pull each subnetwork toward the distribution the whole network has learned, at $\lambda = 1$. It costs no extra forward pass beyond the teacher's, and it matters for the shallow rungs, whose cross-entropy alone would carry a fraction of a percent of the gradient budget. The sampling is unbiased, so the expected objective is the mixture of all depths that a joint loss would compute, at the cost of one path.

The surprise is that the ladder helps the full model too. In matched pairs trained for the same number of steps, the ladder-trained network reached a validation loss of 1.958 at full depth against 2.037 for the same architecture trained without a ladder, and its 16-layer exit matched the control's full 18 layers. Asking the stack to be useful at every prefix is a regulariser: no single block can become load-bearing, and the early blocks are pushed to do real work rather than defer it.

What each depth scores

The ladder is only interesting if the subnetworks are models rather than curiosities. Below are the four shipped depths of the Needle 3 file on two tool-calling suites, sliced from one checkpoint and run through the same engine with the confidence gate on.

The 2-layer subnetwork is missing from that chart for an honest reason: at 2 layers the base model's confidence head withholds almost every call, so its gated score is zero, and even with calls forced it lands at 20.5 on DroidCall. Two layers of a generalist is not a product. Two layers fine-tuned on one product's tools is a different matter, which is the next figure.

Fine-tuning lifts every depth by 18 to 36 points on DroidCall, and from 4 layers up the tuned subnetwork passes DeepSeek V4 Flash. The 2-layer model goes from unusable to 56.5. Constraining a small network to a narrow task is what lets it reach frontier accuracy on that task, and the ladder is what lets you choose how small.

Fine-tune once, deploy any depth

Because the subnetworks share weights, a fine-tune is a fine-tune of all of them. The adapter is trained on the frozen base at the full 20 layers, merged, and then sliced:

$\theta_d = \mathrm{slice}\big(\theta + BA,\ S_d\big),$

where $BA$ is the low-rank update and $\mathrm{slice}$ keeps the blocks in $S_d$, the engram tables of their sites, and the rows of the confidence head that belong to them. One detail bit us and is worth passing on. A sliced model has its own block numbering, and if you re-slice it using that numbering you get a different, untrained set of blocks; of the 14 subnetworks of a 16-layer slice, 13 differed from the parent's. The fix is that every slice carries its parent's selection order with it, so that slicing a slice is the same as slicing the original. In the Python package that is a flag:

needle finetune data.jsonl --epochs 10 --out adapter.safetensors
needle build --lora adapter.safetensors --layers 8 --platform linux-arm64 --out ./pi

The same ladder exists along width, with each sampled subnetwork a depth and a channel prefix of every matrix, so the training recipe generalises to two axes; the shipped Needle 3 exposes depth.

What it costs

Sampled steps are cheaper than full steps, so ladder training adds nothing to wall-clock time per step; the price is that 20% of steps update only part of the model, which the self-distillation term recovers. At inference there is no cost at all: a subnetwork of depth $d$ is exactly a $d$-layer model in FLOPs, in cache and in file size, from 9 MB at 4 layers to 29 MB at 16 in CQ2-bit. The blocks are the same bytes at every depth, so a device can hold the full file and choose its depth per request, a fast shallow pass for a simple command and the full stack for a hard one, without a second download. That is the part we have not shipped yet, and the part we are most looking forward to.