Needle 2 is a 14 MB model. It can run in a browser, on a Raspberry Pi, or inside an app without a server. But a model this small has less room to recover from a poorly designed environment. Vague names, overlapping functions, bloated prompts, and loose schemas all show up in the result.
We tested 6,120 runs across six environments: smart home control, customer support, field service, email, meetings, and expense approval. Each environment had separate calibration, validation, and hidden test sets. We changed descriptions, constraints, tool order, and the amount of overlap between tools.
The tests produced a set of rules that now shape how we build Needle environments.
Count the whole context
Before changing the model, count what you are asking it to read. The context includes the system prompt, every tool schema that reaches generation, the user request, and the output budget.
Use Needle's tokenizer for the count. Character estimates are fine for catching obvious problems, but they are not a substitute for the real tokenized input.
Leave room for the answer. A prompt that fits into the context window with ten tokens to spare does not fit once the model starts generating a call. If an environment is too large, shorten the system prompt, reduce the tool set, or shortlist tools before generation.
Keep the system prompt short
The system prompt should contain rules that the schema and executor cannot express. It should not repeat the tool descriptions, list every available target, explain the product, or carry a workflow manual.
A useful starting point is plain:
Use only the available tools.
Ground arguments in the user's request.
If no tool fits, return no calls.Add a rule only when an evaluation case proves that you need it. Test the prompt again after every addition. Long prompts do not merely cost tokens. They also push the model's attention away from the tool definitions and the user's words.
Keep the tool set narrow
Start with the smallest set that can complete the job. Most of our best environments had three to six tools.
Do not expose every function in your product because it might be useful someday. Extra tools consume context and create more ways to make the wrong choice. If a user is editing an expense report, they probably do not need the entire accounting API in the same prompt.
A good environment has a clear boundary. A device controller can manage lights, locks, shades, temperature, and cleaning. It should not also compose email.
If the application has dozens of tools, split the problem into two stages. First retrieve a short list from compact names and descriptions. Then give generation the complete schemas for that short list. When the runtime cannot use separate retrieval and generation schemas, route by domain in application code or expose a smaller environment.
Give each tool one job
Tool names should describe an action that is visibly different from the other actions in the environment. Prefer familiar action names in consistent snake_case. Framework class names, namespaces, and internal abbreviations make the model learn your codebase before it can answer the user.
This pair is hard to distinguish:
[
{ "name": "handle_case", "description": "Handle a support case." },
{ "name": "resolve_case", "description": "Resolve a support case." }
]This version tells Needle what changes in the outside world:
[
{
"name": "authorize_case_refund",
"description": "Return a stated USD amount for an eligible case. Never use this to ship a replacement."
},
{
"name": "authorize_case_replacement",
"description": "Ship a replacement product for an eligible case. Never use this for a cash refund."
}
]Negative boundaries help when two tools share vocabulary. "Never use this for a cash refund" carries more information than another sentence about resolving customer problems.
Avoid catch-all functions such as handle_request, process_task, or manage_device. In our tests, generic fallbacks sometimes turned a clean refusal into an unrelated action.
Use enums at the right stage
Enums do two different jobs. They help the grammar constrain arguments during generation, but they also become part of the text used to select tools. Those jobs can conflict.
Small, stable choices belong in the schema. Currencies, device states, queues, and operating modes are better as enums than open strings.
{
"name": "configure_lighting",
"description": "Change a room's lights. Never use for shades, temperature, locks, or cleaning.",
"parameters": {
"type": "object",
"properties": {
"room": {
"type": "string",
"description": "Exact room to control.",
"enum": ["kitchen", "living_room", "bedroom", "office"]
},
"state": {
"type": "string",
"description": "Requested light power state.",
"enum": ["on", "off"]
},
"brightness_percent": {
"type": "integer",
"description": "Brightness from 0 to 100, only when the user specifies it.",
"minimum": 0,
"maximum": 100
}
},
"required": ["room", "state"]
}
}Precise descriptions raised exact tool selection from 50.3% to 66.3% in our hidden tests. Adding enums and numeric ranges raised it to 79%. Exact arguments improved more sharply, from 36.7% with descriptions alone to 63% with constraints.
Large dynamic enums need different treatment. Repeating hundreds of customer names, device names, or record IDs across many tools consumes context and makes the tool descriptions less distinct. It can improve argument constraints while making the correct tool harder to retrieve.
When possible, retrieve tools using compact schemas, then add the valid target enums to the selected tools before generation. This keeps the retrieval input clean without giving up constrained arguments. If you cannot separate the stages, pass a smaller target set or narrow the tool environment first.
Enums work best when the value appears in the request or comes from a known set. They are less reliable for abstract judgments such as urgency, sentiment, or risk. Those labels need their own evaluation set and may need a larger model or deterministic rules.
Remove fields you do not need
Every required argument is another chance to fail.
An email action may need a subject and a destination queue. It may not need Needle to also produce a sender type, a prose explanation, an urgency label, and a second classification that no downstream system reads.
Keep the fields that change application behavior. Compute IDs, timestamps, policy decisions, and other deterministic values in code when possible.
Dynamic enums are useful once the tool has been selected. If the interface currently shows five open support cases, pass those five case IDs as the allowed values. This prevents case_505 from turning into 505 and blocks invented targets before execution.
Keep the public schema simpler than the backend API. Needle does not need internal IDs, framework selectors, or fields that your code can derive after the call. Translate the small Needle schema into the larger application request deterministically.
Separate retrieval from generation
Tool retrieval and call generation are different tests.
Retrieval needs short, contrasting descriptions. A tool should be easy to distinguish from its neighbors without reading a long parameter inventory. Generation needs the opposite: exact types, required fields, ranges, and the finite values that are valid for this request.
A practical pipeline looks like this:
1. Describe the available actions compactly.
2. Retrieve the most relevant tools.
3. Attach their complete parameter schemas and current enums.
4. Generate the call.
5. Validate and map the call in code.Do not diagnose both stages with one accuracy number. If the correct tool never entered the shortlist, changing the grammar will not fix the problem. If the correct tool was present but an argument was wrong, a better retrieval embedding will not fix that either.
Keep workflow state outside the model
Descriptions are not a workflow engine.
For a request such as "check the pump, and restart it if it is offline," Needle may emit both calls at once. The second action depends on information the model has not received yet.
The application should enforce the sequence:
1. Run read_asset_status.
2. Inspect the returned state.
3. If the state is offline, expose or approve restart_field_asset.
4. Run Needle again with the result in context.Use the same pattern for permission checks, refunds, replacements, and any action with a precondition. Needle proposes calls. Your executor decides which calls are valid now.
Parallel calls are different. "Turn off the bedroom lights and lock the front door" contains two independent actions, so both can run without waiting for new information.
Reset Needle between unrelated requests. This clears the previous turn while keeping the weights loaded. If a workflow needs another model call, pass the relevant tool result explicitly instead of relying on state left over from an earlier request.
Freeze tool order
Tool order changed exact accuracy by as much as 12 percentage points in our hidden tests. Once an environment passes validation, keep its order stable.
This does not mean there is one correct universal order. It means the order is part of the calibrated environment. Test the original, reversed, and a few shuffled versions, select the best stable layout, and treat later reordering as a model change.
Validate every call
Confidence did not reliably separate correct and incorrect calls in our tests. We saw valid calls at low confidence, false calls at low confidence, and clean refusals at both ends of the range.
Do not use confidence as an authorization policy. Validate the function name, required arguments, enum membership, numeric ranges, current permissions, and workflow state before an action reaches a device or external API.
An empty call list also needs inspection. If generation ended with a token budget error, the result is not a successful refusal just because no complete call was returned.
Log what the model actually saw
Save the exact system prompt, ordered tool schemas, retrieved shortlist, raw model response, and model version for each evaluation run. Without that record, two requests that look identical in a test report may have reached different prompts.
This is especially important when tools or target enums are generated at runtime. A changed inventory can change retrieval and argument generation without any change to the model weights.
The shortlist also tells you where the failure started. If the correct tool was absent, inspect names, descriptions, enum size, and the retrieval set. If it was present, inspect the completion schema and generated arguments.
Test the environment, not a demo prompt
A useful evaluation set should contain ordinary requests, paraphrases, parallel actions, requests that require another turn, and requests the environment must refuse.
Keep a hidden split with names and phrasing that do not appear during calibration. Test tool order separately. Record exact function selection and exact argument values rather than accepting any valid JSON as success.
For larger environments, record each stage separately:
retrieval recall -> selected tool -> exact arguments -> validation -> executionChange one variable at a time. Prompt length, tool names, enums, ordering, and output budget interact with each other. A combined rewrite may improve the final score while hiding which change helped.
Our strongest initial templates were smart home control and expense approval. Atomic field service and customer support actions also worked well. Broad email classification and full meeting transcript extraction were less reliable, so we would narrow those environments before shipping them.
The environment is ready when you can explain a failure at a specific stage. "The model got it wrong" is not enough. You should know whether it missed the tool during retrieval, selected the wrong shortlisted tool, generated a bad argument, failed validation, or ran out of tokens.
Try your own tools in the Needle 2 playground.