Structured Visuo-Semantic Perception

When an agent looks through a camera, it needs the objects, their positions, the writing and the relationships between them. A sentence is one way to carry that information, but composing the sentence is an additional job. The model must choose an order, connect clauses, resolve references and decide how much explanation to give. A small model running on a phone has a limited budget for all of this.

Our argument is that prose construction, verbose semantic descriptions and open-ended reasoning spend capacity that a perception model could use on the image itself. Mirage removes those requirements and introduces structural bias into the problem. It predicts geometry with spatial heads, reads text with a dedicated reader and expresses meaning through compact fields and references. We call this structured visuo-semantic perception.

This article follows that division of work through the released model, from pixels to the record an application receives. The Mirage release post has the runnable examples and device measurements.

Remove the prose, keep the meaning

Consider a model describing a red stop sign mounted on a black pole. A paragraph might introduce the sign, add a clause about its colour, connect the pole with “which” or “and”, then explain where both sit in the scene. An application has to recover the entities and their connections from that paragraph.

Mirage gives the sign and pole names and properties, then records attached_to as a relation between their IDs. The relation still matters. What disappears is the need to compose it into a sentence, maintain a narrative and repeat enough context for a human reader to follow it.

The distinction is between semantic content and its verbal packaging. Removing conjunctions does not mean removing logical conjunction, negation or relationships. Two properties can coexist in a record without a sentence joining them. A relation has explicit endpoints instead of a pronoun whose referent the next component must infer. Open-vocabulary values still allow the model to name and describe things beyond a fixed class list.

The same principle applies to reasoning. Mirage exposes a short generated trace, but its perception interface does not ask for an essay or an unrestricted chain of thought. Decisions can be bounded by a supplied set of choices or a form. A system that needs broader deliberation can use the resulting record as input to an agent.

A red stop sign reading ARRÊT on a black pole, with trees, buildings and grass behind it — Selected outputs from the released 2-bit model on COCO image 231049. The decision is a separate form request on the same photo. Values are model predictions, including the OCR's incorrect accent in ARRÉT.

This example shows selected predictions from the released model. The sign and pole have boxes, masks, names and attributes. Their connection is a relation. The OCR result also shows a mistake, ARRÉT instead of the sign's ARRÊT. A structured result makes the mistake easier to locate; it does not make the prediction correct.

Structure has to reach inside the model

Asking a general visual language model for JSON constrains its presentation. It can still leave the decoder responsible for generating every coordinate, spelling every word and deciding when the inventory is complete. Mirage changes which computations produce those values.

For an image $I$ , the main inference paths can be written as

E, D = \operatorname{encode}(I), \qquad B = \operatorname{detect}(D),

A = \operatorname{masks}(D,B), \qquad R = \operatorname{read}(I,B_{\mathrm{text}}),

M = \operatorname{memory}(E,D,B), \qquad Y = \operatorname{decode}(M),

J = \operatorname{assemble}(Y,B,A,R).

Here $E$ is the coarse image memory, $D$ is the finer spatial feature map, $B$ contains object and text proposals, $A$ contains instance masks, $R$ contains read strings, and $Y$ is the decoder's compact semantic output. The application receives the assembled JSON record $J$ .

These equations describe a computation graph. They do not assign a joint probability to the returned record. A detector score, an OCR character distribution and a decoder token probability describe different predictions, even when all three contribute to one object in the result.

Where the description comes from

Image → shared encoder

512 px canvas · coarse and fine features

Spatial heads

Coarse + fine image features

Object boxes · text boxes · masks

Text reader

Detected crops from the original image

Read strings

Semantic decoder

Image memory + region slots

Names · properties · relations · scene

Runtime assembly → JSON

Resolve references and attach geometry

The spatial heads provide proposals and masks. Proposals guide the reader's crops and the decoder's region slots. The runtime combines their outputs into a record. Read strings are not fed back into the semantic decoder.

The structure enters at three levels. Spatial heads restrict how geometry is produced. Region references restrict how a semantic item attaches to that geometry. The output format restricts how the description is expressed. Reducing prose alone would leave the first two problems with the language decoder.

Geometry on a spatial grid

Mirage fits the image into a 512-pixel square canvas while preserving its aspect ratio. The encoder produces a $16 \times 16$ memory with 512 channels. A feature pyramid combines coarse features with finer stem features to produce a $64 \times 64$ map with 128 channels for the spatial heads. Padding is masked so it does not become image content.

The detector predicts two centre heatmaps, one for objects and one for text. At grid cell $(i,j)$ , each kind has a score, a width and height, and two centre offsets. In grid units, a proposal is

p_{ij}=\sigma(h_{ij}), \qquad c_x=j+\delta_x, \quad c_y=i+\delta_y,

b_{ij}=\frac{1}{64}\left(c_x-\frac{|w|}{2},\ c_y-\frac{|h|}{2},\ c_x+\frac{|w|}{2},\ c_y+\frac{|h|}{2}\right).

The engine keeps local peaks in a $3 \times 3$ neighbourhood, applies the requested score cutoff and limits the result to 64 object proposals and 32 text proposals. It clips boxes to the image extent and maps them back to the original image. Public coordinates use a 0 to 1000 scale on each original image axis.

Representing objects through centres and regressed dimensions follows the approach established by CenterNet, Objects as Points. Mirage uses the two heatmaps to propose regions without assigning an object category at this stage. The decoder supplies the names later.

All grid locations are evaluated in the same head pass. Proposal generation therefore does not need a language model to list one box after another or emit an end token. The locations share encoder features, however, so they are not statistically independent. Local peak selection and proposal limits can still discard objects, especially nearby centres or small regions. The decoder can also omit a proposal when composing the semantic inventory.

Names refer to regions

A box should carry more than its four coordinates into the decoder. Mirage builds a memory slot from the fine feature at its centre, an encoding of its corners and coarse features pooled inside the box.

For proposal $k$ , let $f_k \in \mathbb{R}^{128}$ be its centre feature, $\phi(b_k) \in \mathbb{R}^{64}$ its corner encoding and $\bar e_k \in \mathbb{R}^{512}$ its pooled encoder feature. The slot is

m_k=\operatorname{RMS}\left(W_s[f_k;\phi(b_k)]+W_p\bar e_k\right)+\sqrt{512}\,e_{r_k}.

The corner encoding contains sine and cosine features of the four normalised coordinates. The pooled feature averages valid coarse cells inside the box, including its centre cell. The last term associates the slot with reference token $r_k$ . The decoder attends to the image memory and these slots, then selects reference tokens to attach a name or property to a region.

Only references backed by a proposal are available. For an unused slot,

$\operatorname{logit}(r_k)=-\infty.$

This prevents a reference to an absent slot. It does not prevent the decoder from choosing the wrong existing slot or giving it the wrong name. Grounding is still a semantic prediction. A correct box with an incorrect name is a different failure from an incorrect box with a correct name, and both need to be measured.

Region references have clear precedents. Groma represents proposed regions for grounded language interaction, and ChatRex has its language model select box indices rather than generate coordinates. Mirage uses this separation within a small model with shared image features, dedicated masks and text reading, and a compact perception interface.

Masks without a sequence of pixels

The mask head produces 32 shared prototype maps on a $128 \times 128$ grid. A second branch predicts 32 coefficients at each location of the $64 \times 64$ feature map. For object $k$ , the engine takes the coefficient vector $a_k$ at its selected centre and forms

\ell_k(u,v)=\sum_{p=1}^{32}a_{kp}P_p(u,v), \qquad s_k(u,v)=\sigma\big(\ell_k(u,v)\big).

The object's box restricts the mask's support. The engine thresholds the mask and can return it as a bitmap or polygon outlines. The decoder does not generate the contour coordinates or label each pixel autoregressively.

Shared prototypes combined with instance coefficients are the construction introduced by YOLACT. In Mirage, the prototype branch also receives finer stem features for boundary detail. The benefit of this factorisation is that expensive spatial features are shared across instances. It imposes a limit as well. Before thresholding and box restriction, each mask logit map lies in the span of 32 prototypes. Resolution, prototype quality and the selected box all affect the final boundary.

The instance segmentation article derives both branches and explains why the CPU needs only a $5\times5$ feature patch for each object's coefficients.

The mask and box describe the same selected region, but neither proves the decoder's name for it. The public detection score is a score for the region proposal, not a joint confidence in the name, mask and attributes.

Reading follows the text region

OCR has a different geometry from object naming. Small characters need local detail that may be lost when the whole image is resized. Mirage crops detected text regions from the original image, scales them to 48 pixels high and reads long lines through overlapping windows up to 320 pixels wide. This preserves aspect ratio instead of squeezing every line into one fixed rectangle.

The reader outputs a distribution over characters and a blank at each sequence position. The CTC formulation of Graves and colleagues defines a string probability by summing over frame sequences that collapse to that string,

p(s\mid I_\rho)=\sum_{a\,\in\,\mathcal B^{-1}(s)}\prod_{t=1}^{T}p_t(a_t\mid I_\rho).

The collapse map $\mathcal B$ merges consecutive identical labels and then removes blanks. A blank between two copies of a character preserves a double letter. The shipped greedy decoder chooses the strongest label at each position and collapses the resulting path. That is an approximation to the most probable string, not an exact maximisation of the sum above.

The engine joins the selected window predictions and inserts the read string into the result. It can also include detected text that the semantic decoder did not mention. There is no eight-line inference cap; reading follows the retained text proposals and the requested output sections. The OCR decoding article derives the overlap and shows why CTC collapse must continue across window boundaries.

Read strings are currently assembled alongside the semantic output rather than fed back into the language decoder. This makes an important boundary explicit. The returned OCR may recognise a word that the generated trace misreads or does not mention. A decision requiring exact text should use the OCR result in the surrounding application; the form path does not run the reader.

Compact semantics and bounded decisions

The decoder remains autoregressive,

$p(Y\mid M)=\prod_{t=1}^{N}p(y_t\mid y_{<t},M).$

Its job is narrower than generating a free-form answer. It writes a compact line representation for scene fields, entities, text references, relations and events. The runtime expands those lines into JSON and resolves region references into geometry. Names and property values remain open vocabulary. The structure is fixed enough for the application to consume without recovering it from prose.

This still leaves real sequence problems. The decoder chooses which semantic items to mention and when to stop. It can repeat items, omit a visible object or assign an incorrect relationship. Moving geometry into a head removes coordinate generation from that sequence; it does not remove the sequence itself.

Forms constrain the task further. An application can request named fields with enum choices, bounded counts, short text or a box linked to a preceding choice. The form path is a separate request mode on the same model, rather than an additional unrestricted answer appended to a description.

For an enum candidate $c$ , the engine scores its token sequence together with the field terminator. Given the image memory and preceding form fields $h$ , the score and candidate probability are

\ell(c)=\sum_{t=1}^{|c|}\log p(c_t\mid c_{<t},h), \qquad P(c\mid\mathcal C,h)=\frac{\exp\ell(c)}{\sum_{c'\in\mathcal C}\exp\ell(c')}.

Here $c$ includes the terminator and $\mathcal C$ includes abstention. Scores are not divided by candidate length. These probabilities are conditional on the supplied alternatives and their wording. Changing the choices changes the question and its probability distribution; a value of 0.8 is not an established 80% real-world correctness rate.

The short reasoning trace is also generated text. It exposes some observations and inferences, but is not a verified account of the internal computation. Its useful boundary is brevity and proximity to the perception task. Mirage does not need to solve an open-ended reasoning problem before returning a box or mask.

What the structure saves

There are two distinct budgets. The first is representational capacity. A fixed spatial head and a region-reference interface limit the family of computations the model must express through language. This is our reason for imposing the bias. The second is inference work. Coordinates, mask contours and OCR strings no longer all consume decoder steps, and compact fields avoid sentence construction and repeated connective language.

A useful latency decomposition is

T=T_{\mathrm{encode}}+T_{\mathrm{heads}}+T_{\mathrm{read}}+T_{\mathrm{prefill}}+ \sum_{t=1}^{N}T_{\mathrm{decode},t}+T_{\mathrm{assemble}}.

Reducing $N$ reduces sequential decoding work. The remaining terms still matter. More text crops increase reader work, masks have their own cost, and per-token latency depends on context length and the backend. A shorter output is therefore evidence of less decoding, not by itself evidence of a proportional end-to-end speedup or better perception.

Structural bias also restricts what can be expressed. A 64-proposal budget is finite. A missed proposal cannot be recovered merely by selecting a different reference. Shared masks have limited spatial detail. A compact record is suitable for perception and bounded decisions, while a conversational explanation asks more of the system. We chose the interface around what a local agent needs from its camera.

Where this sits among vision models

The boundary is not simply detectors on one side and language models on the other. Grounding DINO already supports open-set, language-conditioned detection. Florence-2 expresses several vision tasks through a sequence interface, including detection and segmentation. Groma and ChatRex explicitly connect proposed regions with language. These systems establish that open vocabulary, structure and grounding can be combined in several ways.

Mirage's design choice is to make a compact perception record the main interface and assign its fields to computations suited to their geometry. HConvGLU reduces the cost of channel mixing inside the encoder; the HConvGLU article derives that operator separately. Structural perception concerns a different level of the design, namely what the model is asked to produce and which part produces it.

Measure the fields and their connections

A useful evaluation must check both the individual predictions and how they attach to one another. Box recall alone cannot establish grounded naming, and correct names cannot establish localisation.

Part of the result	What to measure
Object proposals	Precision and recall at stated IoU thresholds and proposal budgets
Grounded names	Name correctness on geometrically matched regions
Instance masks	Mask overlap for matched instances, including small objects
Read text	Character and word errors on matched text regions
Attributes and scene fields	Correct, contradicted and missing facts
Relations	Predicate correctness with both endpoints matched
Decisions	Accuracy and probability calibration for a stated choice set
Runtime	End-to-end latency with image size, output sections, precision and device specified

To isolate the capacity argument, a comparison needs matched model budgets and evaluation conditions while varying the semantic format and the division of work. Release latency establishes that this implementation runs at a particular speed. It does not isolate how much of its quality comes from removing prose, using region references or changing the encoder.

Mirage Visual Perception + LLM

If you already have a language model in your application, can Mirage give it enough visual context to answer questions about a photo? This is a useful test of the interface. The perception model has to preserve the facts another model needs, and the language model has to use them without seeing the image.

We tried this with Qwen3-0.6B, leaving both models' weights unchanged. Mirage described each image once. We serialised its record as JSON and passed it to Qwen with a question. No image embeddings, learned connector or fine-tuning were involved.

A text boundary between perception and answering

Image → Mirage
Pixels become scene fields, objects, text and relations.
22 MB vision model
JSON + question
The same visual record can answer several different questions.
Ordinary text tokens
Qwen → answer
A text model reads the record and answers the question.
Unchanged 0.6B language model

Only Mirage sees the image. Qwen receives its predictions as text. No adapter is trained and neither model's weights change.

Let $X$ be the image, $Q$ the question, $P$ Mirage's perception function and $s$ the JSON serialiser. The composition is

J=P(X), \qquad Z=s(J), \qquad p_{\mathrm{system}}(A\mid X,Q)=p_{\mathrm{Qwen}}(A\mid Z,Q).

The language model receives ordinary text tokens. In principle, this boundary can connect Mirage to any text model that can follow the interface. The initial comparison below uses Qwen3-0.6B. It gives the application visual input without making Qwen itself a model that reads pixels.

There is a useful geometric way to think about the limit. Imagine all possible images grouped by the record Mirage assigns them. Two images that become the same $J$ are indistinguishable to Qwen for the same question. A missed skateboard, an unread label or a merged pair of objects cannot be recovered from the record alone. Qwen can guess from its prior knowledge, but that guess is not another observation of the image. Formally, because this perception pass does not receive the question, the data-processing inequality gives

I(A;J\mid Q)\leq I(A;X\mid Q).

The record is useful when it retains the information needed for the question. Structure makes that information addressable. It does not make the record complete.

Composing vision and language through text has a clear precedent in Socratic Models. Our experiment concerns the particular boundary Mirage provides, a small perception model returning named fields, regions and relationships that an existing text model can consume.

The experiment

We ran the experiment locally on 5 November 2026. The final comparison uses 256 questions from 128 COCO images in VQAv2 validation, with two questions per image. We selected images and questions by a fixed hash ordering, before generating their predictions. These images are disjoint from the 128-image development sample used to settle the prompt and answer format.

Mirage saw the image but neither question. It returned scene fields, named objects and regions, descriptions, counts, boxes, scores, OCR, relations, events and the complete generated trace. We omitted masks and points from this text interface. All five conditions reuse the same perception result. We did not repair its names, coordinates, duplicate entities or read strings.

Qwen used its original BF16 weights through MLX on an Apple M5 Max. Thinking mode was disabled, decoding was greedy, and each response had a 64-token limit. Every condition used the same instruction to return a JSON object containing a short answer. We parsed that field with json.loads, without a judge model or response repair. The exact prompt, model revisions, file hashes and dependencies are in the protocol and runtime record.

The five inputs were the question alone, the question with only scene, the structured record without reasoning, the record with the complete trace, and the complete record from the next distinct image in the frozen sample order. This last condition keeps real visual descriptions in the prompt while breaking their connection to the question's image.

We used the VQA consensus rule with the reference punctuation, number, article and contraction normalisation applied to every answer. For prediction $\hat a$ , ten human answers $a_1,\ldots,a_{10}$ and normalisation $\nu$ , its credit is

S(\hat a)=\frac{1}{10}\sum_{j=1}^{10} \min\left(1,\frac{1}{3}\sum_{i\ne j} \mathbf{1}\left[\nu(\hat a)=\nu(a_i)\right]\right).

The table reports mean credit multiplied by 100. These are sample scores, not a submission on the full benchmark.

Context given to Qwen	VQA score / 100	Median total prompt tokens
Question only	9.3	77
Scene fields	28.1	109
Structured record, no trace	33.1	526
Structured record with full trace	31.3	586
Another image's record with full trace	22.2	587

The matched record with its trace beat the mismatched record by 9.0 points. A paired bootstrap over images, keeping both questions together in each of 10,000 resamples, gives a 95% interval of 4.9 to 13.2 points. This is evidence that the correct visual context helped on this sample. The mismatched score also shows why the question-only comparison is insufficient. A plausible but irrelevant record can still change the model's guesses and answer format.

The other differences need more restraint. Adding the trace changed the score by −1.8 points, with an interval of −5.0 to 1.2. We did not find a reliable benefit from passing it along. The record with its trace exceeded scene fields by 3.1 points, but that interval also crossed zero, from −1.1 to 7.2. This experiment does not isolate the value of JSON syntax or compare structure against a prose description containing exactly the same facts.

Read the mistakes as well as the score

COCO image 283 used for the question “What drink is shown?” — Three cases selected after scoring to illustrate a useful answer, missing information and a trace-dependent error. The complete fixed sample is available below. VQA credit measures agreement with the reference answers, not confidence.

For the wine photo, Mirage names a wine bottle and glasses. Qwen answers wine from the structured record, while scene fields alone lead to none. The description still contains errors, including the table material and duplicated reflections. A useful answer does not validate every field.

The skateboard photo shows the opposite outcome. Mirage misses the skateboard and returns several cars, people and shoes. Qwen guesses a skateboard from the question alone, then answers a car when given the record and trace. Extra context can be worse than a language prior when it omits the relevant object and introduces distractions.

The television example makes the trace comparison tangible. The record contains a silver antenna, but also calls the television a digital scale. Without the trace, Qwen answers antenna. Adding the trace changes its answer to digital scale. We can observe that change without treating either generated trace as a faithful explanation of the model's internal computation.

Answer format remains a limitation even with valid JSON. Qwen sometimes puts a complete sentence inside answer. For example, Yes, the animals have stripes. receives zero credit against yes, despite agreeing in meaning. The score therefore measures the complete answering interface, including brevity and wording, rather than visual understanding alone. All 1,280 final responses were valid JSON. One question-only reply reached the 64-token limit and was retained.

We encountered this problem during development. An initial prompt containing coordinate definitions sometimes caused Qwen to echo those definitions. A shorter prompt often produced sentences instead of short answers. We retained both development runs, checked the JSON format on the separate release photo, then froze it before running the fresh comparison above. This was an exploration with prompt development, not a preregistered benchmark.

The final sample contains 90 yes/no questions, 35 number questions and 131 other questions. With the trace included, their scores were 61.3, 21.4 and 13.2 respectively. There is considerable room for improvement in detail retrieval and counting. We did not audit overlap with either model's training data, and this table is not a comparison against a native VLM. The result supports a working composition on this sample, not a claim that a text boundary preserves every visual capability.

We also tried larger text-only readers with the same prompt and records. They did not produce consistent gains. Qwen3-4B Instruct improved on the smaller reader, but its gain over answering without visual context remained uncertain. Qwen3-1.7B and Nemotron 3 Nano 4B often failed the answer-format contract. The follow-up record retains every condition, failed response and confidence interval. These runs give us reasons to examine the interface more carefully before making a stronger claim.

Try the connection

The experiment bundle contains the runner, a standalone ask.py, the fixed questions, all observations and responses, and the scoring results. On Apple Silicon, in an environment with Python 3.12, the example runs with

pip install -r requirements.txt
python ask.py photo.jpg "What drink is shown?"

The essential boundary is small. The runnable example passes this context and the question through Qwen's chat template, then calls the unchanged text model.

import json
import cactus

vision = cactus.Mirage()
observations = vision.describe(
    "photo.jpg",
    fields=["bbox", "score", "position", "description", "count"],
)
context = json.dumps(observations, ensure_ascii=False)

On this machine, warm Mirage calls had a median of 77.5 ms on CPU. Qwen's subsequent answer call had a median of 60.0 ms on the Metal GPU with the full trace. These are separately measured stages, excluding downloads and model loading. They are not an end-to-end pipeline benchmark. The median full prompt contained 586 tokens. The 22 MB size belongs to Mirage alone; Qwen's BF16 weights are additional.

The reusable record is the practical attraction. A camera frame can be perceived once, then read by several questions or application components. For $n$ questions about one image, the work has the form

T(n)=T_{\mathrm{perception}}(X)+\sum_{k=1}^{n}T_{\mathrm{language}}(J,Q_k).

That boundary lets an application inspect the observations, decide which fields it needs and change its language model independently. The next test should ask which facts the record consistently loses, and whether a targeted second look can recover them. A larger language model may read a record better, but it still needs the relevant visual evidence to be there.

The full forward pass gives the remaining architecture equations. You can inspect Mirage's output on your own images in the browser playground and use the same interface through the Cactus package.