What Devices Are Supported on Needle

Needle 3 is a model for the devices that do not get one: budget phones, watches, glasses, cameras, hubs, robots, cars and microcontroller-class boards, as well as the laptops and servers next to them. Every target runs the same engine on the same weights, so the question is only which folder to download and which depth to ship. This guide answers both, then covers the runtime surfaces and the air-gapped path.

The platform folders

Every folder holds an engine under 1 MB that loads needle3.cact at start; the engine carries no weights. The Python package fetches a folder and puts the weights beside it, at the full 20 layers or any smaller subnetwork:

pip install cactus-needle
needle build --platform macos-arm64
needle build --platform linux-arm64 --layers 8 --out ./pi
needle build --platform linux-mipsel --layers 2 --out ./camera

The folders are also published on the Hugging Face repo for a direct download, and needle download <folder> --out <dir> copies one without building.

Picking a depth

Every depth from 2 to 20 layers is a trained model, sliced from one set of weights by needle build --layers n, and the same engine runs all of them. The shipped 20-layer needle3.cact is 35 MB; the ladder runs down to about 9 MB at two layers. On a Raspberry Pi 5 decode runs from roughly 400 tokens/s at the top of the ladder to 4,000 at the bottom, and prefill from 1,000 to 10,000. Every response reports prefill_tps, decode_tps and peak_ram_mb, so measure on the device rather than guess.

A smaller depth loses accuracy on the general task and gets it back when fine-tuned to one product's tools: on DroidCall every subnetwork gains 18 to 36 points and from four layers up the tuned model passes DeepSeek V4 Flash. The pattern that works is to develop against the full model in Python, fine-tune once, then build the depth each device class can afford.

From the command line

Native folders ship a runner. Answer one query and exit, or serve HTTP on localhost:

./needle --model needle3.cact --tools tools.json --prompt "dim the living room to 30"
./needle --model needle3.cact --tools tools.json --serve    # POST /complete {"input": "..."}

tools.json is a JSON array of the functions the assistant may call, in the same shape the Python package builds from a decorated function. --system system.txt passes environment facts, --forced bypasses the confidence gate, and --fail-input-overflow refuses a turn that would not fit the context window instead of trimming it.

From C

needle.h exposes needle_init, needle_complete and needle_embed, returning non-negative counts on success and negative values on failure. Tool schemas may use Needle's compact form or OpenAI-style {"type": "function", "function": {...}} wrappers; camel-cased and qualified names are aliased for the model and restored in the returned calls. The native API owns one process-global model and conversation, so run one worker process per fine-tune when several models must stay loaded at once.

In the browser

The wasm folder ships needle.js and needle.wasm, and it is what the sandbox on this site runs: the engine and the archive are fetched once, cached by the browser, and every request after that stays on the visitor's machine. The same files run under Node.

On a WASI host

wasm-component holds a WASI Preview 2 component and its WIT world. It exports cactus:needle/engine@3.0.0 with the same lifecycle as the native API; model bytes are supplied at runtime and each component instance owns one conversation:

load: func(model: list<u8>) -> result<_, failure>;
init: func(system-prompt: option<string>, tools-json: option<string>, tool-index-path: option<string>) -> result<u32, failure>;
complete: func(input: string, max-new-tokens: u32) -> result<string, failure>;
embed: func(input: string) -> result<list<f32>, failure>;
reset: func();

The component is also published to GHCR as a signed OCI artifact, one image per engine generation, tagged with the engine version; oras pull or wkg oci pull fetch it and cosign verify checks the Sigstore keyless signature from the publish-component workflow in the repo.

From Python

pip install cactus-needle covers macOS on Apple Silicon, Linux x86-64 and ARM64 (glibc and musl), and Windows x64 and ARM through wheel-tagged engines fetched on first use. The Python docs are the reference; every other target is reached through needle build --platform.

No network at all

Inference never touches the network. The Python package caches the engine under ~/.cache/cactus-needle/v3/ and needle3.cact beside it, and an air-gapped device only needs those files in place: needle fetch and needle download needle3 pull them on a connected machine, copy them to the same cache path on the device or into the installed needle/ package directory, or point NEEDLE3_LIB_PATH at the library. Install the package itself with pip download on the connected side and pip install --no-index --find-links <dir> cactus-needle on the device, and set HF_HUB_OFFLINE=1 so a missing file fails fast instead of trying to download. For a native folder, copy the folder and the archive; nothing in it phones home.