Porting Whistle

Whistle runs on the same engine and the same .cact container as Needle 3, so a port of Needle 3 already has most of what it needs. This post covers the rest. It is written from the questions people porting and fine-tuning Whistle have sent us, and it assumes Porting Needle 3 rather than repeating it.

What you have to work from is public. Cactus-Compute/whistle holds whistle.cact, the shipped model at 16.9 MB, and checkpoints/whistle.safetensors, its master weights in float32. The engines and needle.h for every platform are in Cactus-Compute/needle3, and the same binary runs both models. The reference model is needle.model.whistle.Whistle in the cactus-needle package, and the container is written by needle/model/export.py. Every number below was measured on engine 3.3.3, the version cactus-needle pins today.

1. Test the encoder through embed, against the quantised reference

Whistle gives a port something Needle 3 does not, a direct view of the engine's internals. needle_embed(NULL, pcm, samples, out, capacity) returns the encoder output, one row of 512 floats per 80 ms frame, and needle.Whistle().embed(pcm) is the same call from Python. Get the encoder right against that before you touch the decoder.

As with Needle 3, the float checkpoint is the wrong oracle. whistle.cact holds its matrices at 2 bits per weight, with the embedding and the six lane maps at 4, and runs int8 activations and an int8 KV cache. On seven LibriSpeech clips the float reference agrees with the engine's encoder output to a cosine of only 0.79 to 0.94. The same reference in float32 with that quantisation simulated agrees to 0.9994 or better on every clip. Frame by frame it stays above 0.999 except for a frame or two at the very end of a clip, where the worst we saw was 0.984. The package has both halves.

import jax, jax.numpy as jnp, numpy as np, soundfile, needle
from dataclasses import replace
from needle.model.whistle import Whistle, load_whistle
from needle.model.quantize import (cq_quantize, configure_deploy, leaf_name,
                                   _is_quant_leaf, _reduces_second_last)

params, cfg = load_whistle("checkpoints/whistle.safetensors")
cfg = replace(cfg, dtype="float32")

def quantise(path, leaf):
    if not _is_quant_leaf(path, leaf):
        return leaf
    name = leaf_name(path)
    bits = 4 if name == "embedding/embedding" or "mhc_phi" in name else 2
    if _reduces_second_last(path):
        return jnp.swapaxes(cq_quantize(jnp.swapaxes(leaf, -1, -2), bits), -1, -2)
    return cq_quantize(leaf, bits)

qparams = jax.tree_util.tree_map_with_path(quantise, params)
configure_deploy(act_bits=8, kv_bits=8)

def level(pcm):
    blocks = len(pcm) // 320
    rms = np.sqrt((pcm[:blocks * 320].reshape(blocks, 320).astype(np.float64) ** 2).mean(1))
    loud = np.sort(rms)[int(0.99 * (blocks - 1))]
    return pcm * (0.1 / loud) if loud > 1e-6 else pcm

pcm, _ = soundfile.read("clip.wav", dtype="float32")  # 16 kHz mono
enc, frames = Whistle(cfg).apply({"params": qparams}, jnp.asarray(level(pcm))[None],
                                 jnp.array([len(pcm)]), True, method=Whistle.encode)
reference = np.asarray(enc[0, :int(frames[0])])
engine = np.asarray(needle.Whistle().embed(pcm)).reshape(-1, 512)

level is the one step the reference leaves out. The engine scales every clip before its front end, by the rule in section 3, and Whistle.encode does not, so apply it to the waveform yourself. Its effect cancels in the per-channel normalisation everywhere except where a band's power falls near the floor inside the log. That is why clean speech barely shows it, a change in the fourth decimal place of the cosine on our clips, and quiet recordings can.

The decoder exposes no logits, so its oracle is the transcript. We ran the engine's decode loop from section 6 in Python over the quantised reference on 35 LibriSpeech clips, and it reproduced needle_transcribe exactly on 31. The other four differ by one word or one comma ("Mental Board" against "mental board", "sharp" against "shirt"), the near ties where the order of a float sum decides.

So the acceptance test for a port is three checks, in this order. The tokenizer decodes the engine's ids to the same text. The encoder output has the same frame count as embed and agrees with it to a cosine above 0.999 over the clip and on every frame but the last few. The transcript and the language match needle_transcribe on your clips, with the language detected and forced, and with and without keywords.

2. The shipped file is the public checkpoint, packed

needle whistle build writes every matrix at 4 bits, which is why a fine-tuned Whistle comes out at 29.3 MB, and the packer in export.py refuses any other width. So the public tooling as it stands does not reproduce whistle.cact. The file itself is reproducible, though. It is the public checkpoint packed by the same code at the shipped widths, CQ4 for embedding and the six mhc_phi lane maps and CQ2 for every other matrix.

import numpy as np
from needle.model import export
from needle.model.whistle import load_whistle
from needle.model.whistle_finetune import WhistleTokenizer
from needle.model.quantize import _cq_codebook_np, _cq_hadamard_np

def pack(w, bits, group):  # export._cq_pack without its 4-bit guard
    rows, cols = w.shape
    g = np.pad(w, ((0, 0), (0, -cols % group))).reshape(rows, -1, group) @ _cq_hadamard_np(group)
    norm = np.sqrt((g ** 2).sum(-1, keepdims=True))
    idx = export._nearest_idx(g / np.maximum(norm, 1e-12), _cq_codebook_np(bits, group))
    return export._pack_lsb(idx.reshape(rows, -1), bits), norm[..., 0].astype(np.float16)

quantise = export._q
export._cq_pack = pack
export._q = lambda name, mat, bits, group: quantise(
    name, mat, 4 if name == "embedding" or "mhc_phi" in name else 2, group)

params, cfg = load_whistle("checkpoints/whistle.safetensors")
tokenizer = WhistleTokenizer(languages=cfg.languages).blob()
export.write_export(params, cfg.needle(), "rebuilt.cact", tokenizer=tokenizer, audio=cfg)

That gives the same header, the same 681-record directory and the same tokenizer, and 16,919,405 of the file's 16,919,407 bytes. The two that differ are one byte of norms in engram0.tables and one byte of codes in engram1.tables, a weight group sitting on the edge between two codebook entries where the order of a float sum tips it. So checkpoints/whistle.safetensors is the exact master of whistle.cact, and the quantised reference in section 1 is a float32 copy of the shipped weights, up to the same rounding.

3. The front end, as the engine runs it

An earlier version of the launch post described the features as band-limited to 250 to 3500 Hz. They are not. That band only feeds the silence gate in section 8, and the features use all 80 mel bands. In order:

1. Samples that are NaN, infinite or of magnitude 65536 or more read as zero.
2. Level gain. Cut the clip into non-overlapping blocks of 320 samples, take the RMS of each, sort them, and take the one at index floor(0.99 × (blocks − 1)). If it is above 1e-6, scale the whole clip by 0.1 / rms, otherwise leave it alone. This is a selection, not an interpolated percentile, so np.percentile is close but not exact.
3. Frames. There are samples // 160 of them, and frame t reads samples 160t − 200 to 160t + 199, with zero outside the clip. Frames are centred with zero padding, not reflect padding.
4. A symmetric Hann window of 400, 0.5 − 0.5 cos(2πi / 399). torch and librosa default to the periodic one.
5. A 512-point real FFT of the windowed frame and its power spectrum, re² + im², over 257 bins. No magnitude, no square root.
6. The mel filterbank ships in the file as record 679, a float32 matrix of 257 rows by 80 columns. Read it and multiply, rather than rebuilding it.
7. log(x + 2⁻²⁴), the natural log with the floor inside it.
8. Each mel channel is normalised over the clip as (x − mean) / (std + 1e-5), with the sample standard deviation, divided by frames − 1.

Then the stem. Treat the features as an image of frames by mel bands with one channel. The first convolution is 3 × 3 at stride 2 in both axes with one zero of padding on every side, one channel to 128, followed by SiLU. Each of the two later stages is a depthwise 3 × 3 at stride 2 with the same padding, then the 128 × 128 pointwise matrix, then SiLU, with no activation between the two. Mel bands go 80, 40, 20, 10 and frames halve the same way, rounding up. Each frame is flattened channel-major, index 10c + f, and projected from 1280 to 512. Thirty seconds is 3,000 feature frames and 375 encoder frames. None of the convolutions has a bias, and each stored 3 × 3 kernel is a (9, 128) matrix whose row is 3 × time + mel.

4. Where each config value lives

The header describes the decoder exactly as it describes Needle 3. For the shipped file that is a vocabulary of 8199, which is 8192 text pieces and seven language tags, width 512, 8 query heads of 48 over 2 key-value heads, value heads of 64, 8 layers, RoPE theta 100000, a cap of 320 positions, a Hadamard size of 512, 4 lanes, three causal conv taps on Q, K and V, engram sites 3 and 7 with orders 2 and 3 over 18432 slots, and an int8 KV cache. A sliding window of 0 and an empty global mask mean every decoder layer attends over the whole transcript, where Needle 3 reads a window.

The speech side is a record of 13 float32 values, audio.manifest at position 237. Its presence is what marks the file as a speech model.

Everything else is derived, and these are the places a reader goes wrong.

• The encoder takes its width, head counts, head sizes and RoPE theta from the header, but has no conv taps. The header's three taps belong to the decoder.
• Cross-attention uses 8 heads for queries, keys and values alike, with no grouping, so its key projection is 384 × 512 rather than 96 × 512.
• The languages are the tokenizer's last seven pieces, <|en|> to <|pl|> in id order. <s> and </s> are ids 2 and 1, from the tokenizer header.
• The encoder frame is hop × 2^stages, 80 ms, which is also the resolution of word times.

The checkpoint's config needs the same care. Its text field is the text vocabulary, not the embedding's 8199 rows, and it names neither the languages nor the head count. The reference fills both in from code, seven languages and dim / v_dim heads, so a reader that takes text as the vocabulary, or dim / qk_dim as the head count, loads the wrong model. Its dtype field says bfloat16 because that is what it trained in, and the tensors are stored in float32.

5. The tensor order

The contract is the same as for Needle 3. The directory is positional, and the order below is what tag 0x05E12A84 promises for a speech model. A port should assert all 681 records and refuse anything else.

The ten cross-attention tensors of a decoder layer are cross_norm, q_proj (384 × 512), k_proj (384 × 512), v_proj (512 × 512), gate_proj and out_proj (512 × 512), cross_q_norm and cross_k_norm (48), post_cross_norm and cross_gate. The 43 of an encoder block are its six norms in the order pre_hada_norm_0, ZCRMSNorm_0, post_attn_norm, conv_norm, conv_out_norm, pre_hada_norm, then q_proj (384 × 512), k_proj (96 × 512), v_proj (128 × 512), gate_proj, out_proj, q_norm, k_norm, attn_gate, pw1 (1024 × 512), pw2 (512 × 512), dw (FP16, 9 × 512), and last the 13 tensors of hadamard_mlp_0 followed by the 13 of hadamard_mlp, each in the order d1, d2, b2, d3, d4, w1a, w1b, w2a, w2b, w3a, w3b, cond_v, cond_u. There are no probe heads and so no heads.manifest. Matrices are stored [out, in], as in Needle 3.

6. The blocks and the decode loop

The encoder block is a macaron. The two Hadamard MLPs take half steps around the attention and the convolution module.

x = x + 0.5 · hadamard_mlp_0(pre_hada_norm_0(x))
x = x + σ(attn_gate) · post_attn_norm(self_attn(ZCRMSNorm_0(x)))
h = pw1(conv_norm(x)),  h = h[:512] · σ(h[512:])
x = x + pw2(silu(conv_out_norm(depthwise_9(h))))
x = x + 0.5 · hadamard_mlp(pre_hada_norm(x))

The depthwise convolution runs over time with four zeros of padding on each side and applies the stored kernel as it is, without flipping it. Encoder attention is bidirectional over the clip's frames, with RoPE on Q and K after their norms.

The decoder block is Needle 3's, with a cross-attention step after the self-attention.

x = x + engram(x)                                                    at layers 3 and 7
x = x + σ(attn_gate) · post_norm(self_attn(norm_in(x)))              causal, RoPE, 3 taps
x = x + σ(cross_gate) · post_cross_norm(cross_attn(cross_norm(x), memory))
x = x + hadamard_mlp(pre_hada(x))

Both stacks run inside four mHC lanes exactly as Needle 3 does, and each ends with the mean over lanes and its final norm. Cross-attention normalises Q and K per head, uses no RoPE, scales by 1/√48, and multiplies its output by σ(gate_proj(x)) before out_proj. Its memory is the encoder output plus a gated sinusoid, enc + σ(pe_gate) · [sin(tω), cos(tω)] with ω_i = exp(−i · ln 10000 / 255) for i from 0 to 255, sines in the first half of the width and cosines in the second. The shipped pe_gate is −6.11, so the term is small, but it is there. Each layer projects K and V from the memory once per clip.

The prompt on the wire is two tokens. Feed <s>. Unless a language was forced, the language is the argmax of that step's logits over ids 8192 to 8198. Then feed that tag, and decode until </s>. The decoder input is the embedding row times √512, and the output head is the same embedding.

Decoding is a beam search over five beams.

• At each step every live beam proposes its six best text tokens, ids below 8192 with </s> among them. Each proposal is scored as the beam's running log probability plus the token's, normalised over the text tokens only.
• All proposals are sorted together and taken in order until five beams are live. A proposal that ends the transcript, by </s>, by a loop or by reaching 319 tokens, finishes instead of taking a beam, and is scored as its log probability divided by its token count. The best finished score wins.
• The search stops once five transcripts have finished or no beam is left.
• A loop is the same unit of up to eight tokens repeated four times at the end. The transcript is cut back to one copy of it.

Greedy decoding gives you a correct model and mostly the same text. The beam gives you the engine's text.

The ladder is on the decoder only, with the same endpoint-preserving bisection as Needle 3. Over eight layers the order is 0, 7, 3, 5, 1, 2, 4, 6, and depth N keeps the first N, so depth 4 runs layers 0, 3, 5 and 7. An engram site fires only when its layer is active. --audio-depth N picks a depth at load time and needle whistle build --layers N writes a shorter file. The encoder always runs all eight blocks.

7. Keyword biasing

Keywords reach the engine as one string with one phrase per line. A Python list is joined with newlines, and a single string with commas in it is one phrase. For each phrase the engine takes three spellings, as given, lower case and title case, and tokenises each twice, with a leading space for use inside the transcript and without one for use only as its first word. Spellings that contain a special token are dropped, and the rest go into an Aho-Corasick automaton.

Each beam carries its own position in the automaton. A token that extends a match gets +2.0 on its log probability for every token matched so far, and a token that breaks the match pays the accumulated bonus back. So a phrase of n tokens that completes ends 2n ahead, and a phrase abandoned halfway is worth nothing. The bias only touches candidates the beam search scores, so a port that decodes greedily without the automaton sees no effect at all.

On the shipped engine, on LibriSpeech clips with names the model has not met:

Of eleven clips, six moved to the keyword, two were already right without it, and three did not move, among them "Kaliko" heard twice as "calico". It is a bias, not a constraint, and the "Irolg" row shows that it can rewrite a neighbouring word as it pulls the name in.

8. The gates around the model

Silence. The engine measures the clip before decoding. Per frame it sums the power in the mel bands whose filters lie within 250 to 3500 Hz and takes it in dB. If the 99th-percentile frame, by the same selection rule as the level gain, is less than 5 dB above the 10th, the call returns an empty text and an empty language without decoding. Digital silence and steady noise stop here. Other sounds do not, which is why a voice activity detector belongs in front of an always-on microphone.

Long audio. Up to 30 seconds is one pass. Longer audio runs as 30-second windows. Each window is cut at its last sentence end that falls at least 40% of the way in and half a second before its end, or failing that at the last word ending a second before its end, or else at its end, and the next window starts at the cut. Word times count from the start of the clip.

Streaming. needle_stream_transcribe_process takes about a second of audio at a time, re-transcribes the buffer and commits the words that two consecutive passes agree on.

Word times. They come from a second, teacher-forced pass over the chosen transcript. The engine collects the cross-attention weights of the active decoder layers in the upper half of the stack, standardises each head's weights per frame across the tokens, sums the heads, and finds a monotonic path from tokens to frames by dynamic time warping. A word starts at a piece that begins with a space, its times are frames × 80 ms, and its probability is the mean of its tokens'. Port this after the transcripts match.

Community runtimes

If you publish a port of Whistle, tell us at founders@cactuscompute.com and we will link it here and from the model card. If you are fine-tuning rather than porting, Fine-tuning Whistle has the commands. If a note here turns out to be wrong for your implementation, that is a bug in the notes, and we would rather fix it than have you find it twice.