The feed-forward network holds two thirds of a transformer's non-embedding parameters. A decade of interpretability work says those layers are the model's memory: key-value stores over training facts, the place where editable associations live. Nobody had run the converse experiment, delete them and measure what is lost, with the confounds controlled. We did, and Needle 2 and Needle 3 are built on the answer. The paper is A Controlled Study of Attention-Only Transformers; this post is the intuition.
The block
A Simple Attention Network, SAN, is a decoder transformer with the feed-forward sublayer removed from every block. Everything else is standard: pre-norm, grouped-query attention, rotary positions, a scalar gate on the residual write, tied embeddings. The control arm is the same code with a SwiGLU feed-forward network put back after attention.
Removing the feed-forward layer removes the only learned per-position feature map. What remains nonlinear is the attention softmax and the normalisations, and that has a consequence you can state exactly: for a fixed attention pattern, a SAN layer is linear in its inputs, and each head's update at a position lies inside the convex hull of the value vectors of the positions it attends to. A SAN layer selects and transports content that is present in context. It cannot synthesise a representation the context does not support. That is the precise sense in which the model is context-grounded, and every result below is a measurement of what that restriction costs.
Three ways to ask the question
Deleting the FFN perturbs three things at once: parameter count, compute per token, and the number of nonlinear composition steps. Fix any one axis and the comparison is confounded on the other two, so we ran it three times.
| Arm | Configuration | Total params | Non-embedding | GFLOPs / token |
|---|---|---|---|---|
| SAN | 20 layers, d=512, no FFN | 24.13M | 15.74M | ~0.40 |
| FFN, matched parameters | 4 layers, d=512, FFN 2048 | 24.12M | 15.73M | ~0.20 |
| FFN, matched FLOPs | 9 layers, d=512, FFN 2048 | 43M | 35M | ~0.39 |
| FFN, matched depth | 20 layers, d=512, FFN 2048 | 87.06M | 78.67M | ~0.72 |
Every arm got its own learning-rate sweep, with a boundary rule that extends the sweep whenever the edge wins, so no architecture competes at another's preferred rate. That mattered: the optimal Muon rate differs by 2x between the arms, and a shared rate silently biases the whole comparison. We also trained a pair of models that differ only by a mathematically optimizer-equivalent normalisation parameterisation, to calibrate the noise floor of the pipeline: 0.0015 nats. Models were pretrained on SYNTH, a reasoning-dense corpus with query, trace and answer regions, for up to 105B tokens, and eight predictions were registered before the measurements were made.
What the FFN costs, in nats
The three numbers order exactly by how much of the parameter budget the control lets attention reclaim. Delete the layers in place and you delete 72% of the model, which costs 0.47 nats. Match compute and the standard transformer still leads by 0.26, because attention spends part of its budget on the parameter-free quadratic term and so carries fewer weights at equal cost. Match parameters, reallocating the freed budget into attention depth, and the gap is 0.0055 and 0.0054 nats on two clean seed pairs, 0.27% of loss, agreeing to one part in ten thousand.
The FFN's parameters matter. Its functional form, on this distribution, largely does not.
Two more axes say the same thing. The matched-parameter gap shrinks with training: 0.046 nats at 5B tokens, 0.019 at 30B, 0.0055 at 105B, each budget trained and tuned separately rather than one curve read at three points. And it does not grow with scale: at a fixed 31.5B-token budget across five matched size pairs the attention-only model wins at the smallest size, then the gap plateaus near 0.02 nats from 16M to 57M non-embedding parameters. Constraining both arms to 2M, 8M or 32M unique documents, up to 18 epochs, costs at most 0.010 nats and shows no interaction with architecture.
Where the gap lives
A small aggregate gap could be a small deficit everywhere or a large deficit somewhere. Every SYNTH document has three regions, a query, a reasoning trace and an answer, delimited by marker tokens, so the loss can be decomposed exactly. The token-weighted sum of region gaps reproduces the aggregate to within 2%, so nothing is hiding between the rows.
The deficit sits on query tokens. These are the positions with the least context to route from, where only knowledge stored in weights can help, and at 31B tokens the per-token gap there is five times the sample aggregate while those tokens carry 8% of the loss. By 105B the localisation is complete on this sample: the SAN leads on traces and on every answer region, including memorisation exercises, and trails only on queries. Across the size ladder the query deficit is the one invariant, positive at every size and budget, while every other region changes sign.
Benchmarks split the same way. Lambada asks for a specific content word and, for models trained on SYNTH, is an out-of-distribution recall task; it favours the FFN model at every budget. Sciq puts the answer in a support passage; it favours the SAN, and the margin grows with training in the direction we registered in advance, 0.725 to 0.742 for the SAN while the FFN model slides from 0.702 to 0.661.
That account makes a prediction about text we had not trained on. Natural web text is mostly low-context prediction, so we registered, before launching the run, that a matched-parameter pair trained on fineweb-edu would show a gap between 0.02 and 0.05 nats. It measured 0.040. The same pair reverses on lambada, 0.203 against 0.181 in the SAN's favour: trained on distribution-matched text, the passage is enough to infer the final word and lambada becomes a routing task. Whether a task is storage or routing is not a property of the task. It is a property of the match between the task and the training distribution.
The mechanism, in the weights
Tracking the stable rank of every weight matrix over training gives the same picture in every architecture, size and budget we trained. The routing matrices, Q and K, crystallise within the first quarter of training and do not move again; the optimiser sets the level, with Muon holding them two to three times flatter than AdamW, but the schedule is universal. The matrices that write content into the residual stream keep accumulating rank for as long as the stable phase lasts: the down-projection in FFN models, and in attention-only models the output projection, which inherits the role because it is the SAN's only write path. Selection structure is learned early and frozen; content capacity accrues indefinitely, in whichever matrices can hold it. Representation rank stays high throughout, a minimum layer rank of 173 of 512 at 20 layers, so the classical rank-collapse regime for pure attention is never approached once residuals and normalisation are in place.
What keeps a deep attention-only stack trainable
Component ablations at 20 layers isolate three facts. QK-normalisation is load-bearing: removing it diverges outright at the tuned learning rate, the only divergence in the study and the one finding none of our predictions anticipated. Scalar residual gates are performance-neutral at every depth from 20 to 48 layers and in both architectures; their value was diagnostic, since their trajectories showed the FFN model self-pruning toward attention-only form under learning-rate stress. And post-attention sandwich normalisation is the only variant that beats the baseline, by 0.009 nats. Depth at matched parameters is U-shaped with a 20-layer optimum, and 48-layer attention-only stacks train without incident. Every instability in the study happened in the FFN arm; none in the SAN arm.
When to use one
At matched parameters a SAN pays roughly twice the FLOPs per token at 2048 context, and at matched FLOPs the FFN model is ahead. The trade favours attention-only models where parameters, not FLOPs, are the binding constraint, which is exactly the on-device regime: memory limits set the model size, and a token costs bandwidth before it costs arithmetic. It favours them where a single repeated, interpretable primitive has analysis value, and on trace-rich distributions where the answer can be routed from context. We claim a regime, not superiority.
That regime is Needle's. Tool calling and structured extraction are routing problems: the function names, the argument values and the schema are all in the prompt, and the model's job is to select and transport them into a call. Needle 2 was a Simple Attention Network with an engram memory bolted on for the facts that are not in context; Needle 3 keeps the attention-only stack and adds a ladder so that every depth from 2 to 20 layers is a deployable model. The 0.006 nats above is the price of that design, and the query-token deficit is the part of it we buy back with engrams.
All results are at or below 87M parameters and 105B tokens, on one reasoning-dense corpus plus one knowledge-dense control pair, and MMLU-class benchmarks are at chance at these scales. The storage account itself predicts a wider gap on storage-heavy mixtures at larger scale, which is the natural next experiment. Code, training curves, evaluation reports and every checkpoint accompany the paper.