CTC Decoding Across Image Windows

Imagine photographing a receipt beside a short label. The letters may be about the same height, but the receipt needs much more horizontal space. If we squeeze both text crops into the same rectangle, the long line becomes cramped. We'd like to keep the letters readable while giving the reader a fixed amount of work at a time.

Mirage does this by preserving the line's aspect ratio and sliding a fixed-width window along it. Neighbouring windows overlap so the reader has context near each join. We keep their central frame predictions, assemble them into one sequence, and then apply CTC collapse to get the text. That last ordering matters for a letter that crosses a window boundary, and for repeated letters such as the two o's in “book”.

The same general idea has been used in speech. Hugging Face's 2022 account of long-form recognition describes overlapping CTC chunks and joining their central predictions before decoding. For Mirage's convolutional image reader, we can work through the geometry explicitly. We'll put the frames on a common ruler, measure how far each one can see, and use that to choose where the windows join.

Keep the line readable

The detection head finds text regions on the image canvas. The reader crops those regions from the original RGB image, then resizes each crop to a height of 48 pixels. For a crop of width $w$ and height $h$, the used strip width is

$U=\max\!\left(1,\operatorname{round}\left(48\frac wh\right)\right).$

Geometrically, this gives every crop the same height while preserving its shape. A line twice as wide relative to its height gets twice the horizontal room, up to pixel rounding. The CPU uses nearest-even rounding for $U$ and the detector's normalised-box conversion for the crop coordinates. If you're reproducing the input, both rules are part of the resize.

A reader window is $48\times320$ pixels. Short strips are padded with RGB value 127. Long strips use windows starting at

$x_j=224j,$

giving a 96-pixel overlap. Picture laying a 320-pixel ruler over the strip and moving it 224 pixels at a time. Each new placement shares part of the image with the previous one. We resize the strip once before taking these windows, so every placement refers to the same pixel grid.

The reader's forward pass

Pixels are scaled by $x/127.5-1$. The first $3\times3$ convolution has stride two in both directions and produces 32 channels. Three depthwise-plus-pointwise stages then use channel widths 64, 128 and 384, with vertical-horizontal strides $(2,2)$, $(2,1)$ and $(2,1)$.

For a 320-pixel window, the spatial shapes are

Notice how the shape changes. We compress the vertical dimension more aggressively, until only three rows remain, while keeping 80 positions along the writing. Each stage normalises channels at each location, with SiLU before the next depthwise convolution. Finally, SiLU and an average over those three rows give us an $80\times384$ sequence to read from left to right.

Four residual sequence blocks refine this sequence. With $N_b$ denoting channelwise RMS normalisation and $\phi=\operatorname{SiLU}$, block $b$ computes

$h_b=\operatorname{DW}_{1\times5,b}(N_b(X_b)),$

$[v_b,g_b]=W_{\mathrm{up},b}h_b,$

$X_{b+1}=X_b+W_{\mathrm{down},b}\left[v_b\odot\phi(g_b)\right].$

The up projection produces 768 channels, split into two groups of 384. The down projection returns to 384. A final normalisation and affine projection give character logits,

$Z_t=W_{\mathrm{char}}N_f(X_{4,t})+b_{\mathrm{char}}.$

Each of these 80 frames scores the supported characters and a blank label at index zero. A frame is a position in the image-derived sequence; several frames may contribute to one printed character. Because the reader uses local convolutions and normalises only across channels, we can trace exactly which input columns may affect a frame.

How much context a frame needs

Start with one output position and look backwards through the convolutions. A wider kernel brings neighbouring positions into view. A stride changes how far apart those positions are on the original strip. Keeping track of both gives a recurrence for the receptive-field width $r$ and the spacing $j$ between adjacent outputs,

$r_{\mathrm{next}}=r+(k-1)j,\qquad j_{\mathrm{next}}=sj.$

Starting from $(r,j)=(1,1)$, the four image stages give

$(3,2)\longrightarrow(7,4)\longrightarrow(15,4)\longrightarrow(23,4).$

Each of the four sequence convolutions adds $4\cdot4=16$ pixels. The final receptive field is therefore

$r=23+4\cdot16=87,\qquad j=4.$

We can now draw a frame as a point on the strip with a neighbourhood extending 43 pixels to either side. With the CPU reader's symmetric padding, its centre is at column $4t$, so its input support is the interval $[4t-43,4t+43]$. Adjacent centres are only four pixels apart, and their neighbourhoods overlap heavily. The four-pixel spacing tells us where a prediction sits; the 87-pixel support tells us how much context it can use.

Pointwise projections, channelwise normalisation, gating and averaging over height leave this horizontal support unchanged.

Near an internal window edge, part of that neighbourhood would fall outside the crop. This gives us a geometric reason to overlap windows and discard their edge predictions. Half of our 96-pixel overlap leaves a 48-pixel margin. The first retained frame after a left trim is centred at column 48, with support starting at column 5. The last retained frame before a right trim is centred at column 268, with support ending at column 311. Both neighbourhoods fit inside the 320-pixel window.

The overlap is therefore large enough for the retained interior frames to see their full local context. Reproducing whole-strip outputs still calls for checks of padding at the true line ends, rounding, quantisation and kernel arithmetic. If we change the reader's receptive field or add a global operation, we also need to revisit this construction.

Keep every frame exactly once

Having given each retained frame enough context, we need to join the windows without repeating or skipping a position. Their starts lie on the same four-pixel lattice because 224 is divisible by four. Each window produces $T=320/4=80$ frames, half the overlap is 12 frames, and successive windows start 56 frames apart. Using half-open intervals, we keep

The local frame $t$ in window $j$ sits at global index $x_j/4+t$. The first window keeps $[0,68)$, and the next keeps $[68,124)$. These intervals meet at one boundary. On the pixel ruler, that boundary is column 272, halfway through the overlap from 224 to 320. The next boundary is column 496. At the end of the line, we stop at the last used frame so the grey padding doesn't become extra text.

The intervals tile

$[0,\lceil U/4\rceil)$

without gaps or duplicates. For example, a 1,200-pixel strip uses five windows starting at 0, 224, 448, 672 and 896. Their retained frame counts are 68, 56, 56, 56 and 64, totalling 300.

For greedy decoding, the engine can retain only

$\pi_t=\operatorname*{arg\,max}_c Z_{t,c}$

at each selected frame, including blanks. Keeping the full logit vectors would be necessary for other decoding methods, such as a beam search that scores alternative paths. Pointwise argmax is sufficient for the greedy path used here.

Collapse once, across the seam

We now have a continuous path of frame labels along the line. A letter may occupy several neighbouring frames, so reading every label literally would repeat it. Connectionist Temporal Classification handles this with a collapse map $\mathcal B$ that merges consecutive identical labels and then removes blanks. For example,

$\mathcal B(a,a,\varnothing,a)=aa,$

The blank separates two occurrences of the letter without printing a character of its own. Removing it first would join the two runs and incorrectly leave us with one a.

Window boundaries introduce a second ordering problem. In general,

$\mathcal B(p\mathbin\Vert q)\ne\mathcal B(p)\mathbin\Vert\mathcal B(q),$

where $\Vert$ means concatenation. Imagine a run of a frames stretching across the seam. Each window sees part of the same run, but collapsing them separately can print one a from each half. Deleting matching characters at string boundaries doesn't solve this either. It would erase a real double letter when the blank separating its two runs had already been removed.

The engine joins retained frame IDs first and collapses the resulting path once. An equivalent streaming implementation carries the previous raw frame label across chunks,

previous = 0
text = []
for frame_ids in retained_windows:
    for label in frame_ids:
        if label != 0 and label != previous:
            text.append(charset[label - 1])
        previous = label

The small but essential detail is that previous updates on blanks too. It remembers whether a repeated label continues the same run or starts a new one after a blank. Keep it across windows of the same line, and reset it when moving to another line. That is all the boundary state this greedy collapse needs.

Several frame paths can collapse to the same string. This is worth keeping in mind when interpreting greedy decoding. Mirage collapses the most probable frame path. Finding the most probable string would instead require combining the probability mass of all paths that spell it, which can give a different answer. The engine uses the greedy path without a separate language-model search over OCR strings.

What the window bounds

On the CPU, reader activations and logits are computed one window at a time. Their shapes therefore stay bounded as the line gets longer. Work still grows with the number of windows,

$n(U)=1+\max\!\left(0,\left\lceil\frac{U-320}{224}\right\rceil\right).$

For long strips, the ratio of processed width to unique width approaches $320/224\approx1.43$. We read some pixels more than once so the predictions we keep have neighbours on both sides.

The fixed window bounds an individual reader invocation. We still store the resized strips and the joined frame IDs, so total memory grows with those input and output lengths. A backend that batches several windows also needs space for the batch.

Checks that catch boundary mistakes

We checked the frame-selection rule for every used width from 1 through 10,000 pixels. Each produced exactly the expected global frame indices, including partial last frames and the transition from one window to two.

We also enumerated paths of up to eight frames over two characters and blank, checking every two-chunk split. Carrying the previous raw label gave the same result as a single collapse for all 63,972 splits. These small cases let us check the seams directly, including repeated letters and blanks. Recognition quality on photographs is a separate measurement.

When you try this on an image, the crop remains a good place to start looking for errors. The reader needs the full text region, enough detail to distinguish the characters, and an alphabet that includes them. Joining windows correctly won't recover a letter cut off by the detector.

In Mirage's structured perception record, the reader supplies the text attached to a region reference. The language decoder does not need to reproduce every character through its own token stream. The instance mask head uses a similar separation for contours. You can try both in the Mirage playground or through the Cactus package.