HConvGLU for Channel Mixing in Image Encoders

Mirage's image encoder uses 512 channels to describe each cell of a $16 \times 16$ image grid. Mixing those channels with a conventional feed-forward layer of hidden width 2048 would require about 2.1 million weights per block. Repeating that across the encoder would spend much of the model's storage on channel mixing alone.

HConvGLU uses a fixed Walsh transform to put each feature vector into another basis, mixes nearby coefficients with learned bands, then transforms back. A GELU gate makes the result depend nonlinearly on the input. The released encoder uses 28 of these mixers, each with 100,352 parameters. This article derives the forward pass, explains the restriction imposed by the bands and counts the work the operator actually performs.

What a dense mixer spends

For a column vector $x \in \mathbb{R}^{d}$, a two-matrix feed-forward layer with expansion ratio four has the form

Ignoring biases, it stores $8d^2$ weights and uses $8d^2$ multiply-accumulates per vector. At $d=512$, both counts are 2,097,152. These are the costs of this specific dense baseline, not of every possible feed-forward layer or a matched GLU.

A channel mixer operates within one feature vector. Spatial convolutions and attention move information between image locations. Keeping those jobs separate lets us ask a narrower question about the channel map. How much freedom do we need to give every pair of channels, and how much can a fixed transform supply?

The Hadamard MLP in Needle learns structured Kronecker mixing factors. HConvGLU keeps the basis fixed and spends its learned parameters on bands inside that basis. The two constructions share a connection to Walsh-Hadamard transforms, but they define different families of functions.

Gabor initialisation supplies a separate spatial bias to the encoder's convolutions. It changes their starting weights. HConvGLU defines the channel operator used throughout inference.

The exact Walsh basis

For $n=2^q$, the normalised Sylvester construction is

Every entry is $\pm n^{-1/2}$ and the rows are orthonormal. The sequency of a row is its number of sign changes when read from left to right. Sorting the rows by that number gives the sequency-ordered matrix $W$ used in HConvGLU.

For an explicit indexing rule, let $\operatorname{rev}_q$ reverse the lowest $q$ bits and let $\operatorname{popcount}$ count set bits. With indices starting at zero,

The symbols $\oplus$, $\&$ and $\gg$ denote bitwise XOR, AND and right shift. Row $j$ of this matrix has exactly $j$ sign changes.

With this ordering, $W$ is symmetric as well as orthonormal,

$W^T=W, \qquad W^TW=I, \qquad W^{-1}=W.$

The same operation therefore maps a vector into the Walsh basis and back. An arbitrary permutation of an orthogonal matrix's rows would remain orthogonal, but would not necessarily remain symmetric. The ordering is part of the operator, so a port should not substitute a natural-order transform without accounting for the permutation.

Sequency orders the patterns by sign changes. It does not make neighbouring rows close in Euclidean distance. Distinct normalised rows remain orthogonal, at distance $\sqrt2$. Nor does channel index represent a spatial direction in the image. Locality along the sequency axis is the bias we impose on the channel mixer.

A band in the transformed coordinates

Let $u=Wx$. For an odd band width $k=2r+1$, define

The weight array has shape $n \times k$. Its first index is the output channel and its second index is the tap. Every output has its own weights. Despite the name HConvGLU, this is not a translation-invariant convolution with a single kernel shared across the sequency axis. The edges are zero-padded, not circular.

Mirage uses $n=512$, $k=65$ and $r=32$. Each output can mix a neighbourhood of 65 transformed coefficients, including itself. The matrix is banded in these coordinates, while the equivalent linear map in the original channels is $WBW$.

A sparse band can produce a dense channel map because the basis vectors each span all input channels. This is a possibility, not a guarantee for every choice of weights. For example, $B=I$ gives $WBW=I$. The change of basis also preserves rank,

$\operatorname{rank}(WBW)=\operatorname{rank}(B).$

A full-rank band therefore gives a full-rank channel map. Full rank does not mean unrestricted expressivity. The linear family $\{WBW\}$ has only as many degrees of freedom as the active entries of $B$, rather than $n^2$. HConvGLU accepts that restriction and adds an input-dependent gate.

The gated forward pass

The complete mixer has three bands and one bias,

Here $\odot$ is elementwise multiplication and $\phi$ is GELU. This multiplicative construction follows gated linear units and the GELU-gated variant described by Shazeer. HConvGLU supplies the fixed transforms and banded projections around that gating operation.

The forward implementation uses the usual tanh approximation to GELU,

GELU is not a sigmoid gate. Its output can be negative or greater than one. It modulates the value branch rather than assigning it a probability.

The gate is also why HConvGLU cannot be reduced to one fixed matrix. At a particular input, its Jacobian is

The two diagonal factors depend on $x$. They alter the effective channel mixing for each input while the three bands constrain the learned projections. This gives a more precise picture of the tradeoff than saying the transform makes a small layer equivalent to an arbitrary dense one.

For a width $d$ that is not a power of two, let $P$ pad to $n=2^{\lceil\log_2d\rceil}$ and $C$ keep the first $d$ output coordinates. The general expression is

Mirage uses $d=n=512$, so padding and cropping are unnecessary.

Computing the transform

A dense matrix multiplication is unnecessary for $W$. The following NumPy implementation uses the same sequency butterfly as Mirage. It accepts any leading batch dimensions and transforms the last axis.

import numpy as np


def walsh_sequency(x):
    x = np.array(x, copy=True)
    n = x.shape[-1]
    if n < 1 or n & (n - 1):
        raise ValueError("the last dimension must be a power of two")
    width = 1
    while width < n:
        blocks = x.reshape(*x.shape[:-1], n // (2 * width), 2, width)
        left = blocks[..., 0, :]
        signs = np.where(np.arange(width) % 2 == 0, 1, -1)
        right = blocks[..., 1, :] * signs
        x = np.stack((left + right, left - right), axis=-1).reshape(x.shape)
        width *= 2
    return x / np.sqrt(n)

Each stage interleaves sums and differences, with alternating signs on the right branch. There are $\log_2n$ stages and $n$ additions or subtractions per stage. A normalised transform also needs the final scale, unless it is folded into adjacent weights.

A useful port check is to compare the butterfly with the explicit matrix above, then check $W(Wx)=x$ within the chosen floating-point tolerance. Testing only the inverse property would miss a different self-inverse ordering. The band operation also needs an edge test so an out-of-range tap cannot wrap onto the opposite end.

Counting storage and arithmetic

The archive stores all $nk$ taps of each band. At the boundaries, some of those taps multiply zero and do not affect the result. For $r<n$, the number of active coefficients in one band is

The two edges omit $1+2+\cdots+r$ coefficients each. At $n=512$ and $k=65$, there are 33,280 stored tap slots and 32,224 active coefficients per band.

The bias requires 512 additions. Two transforms require another $2n\log_2n=9,216$ additions or subtractions. GELU, the elementwise product and any transform scaling are additional work. Normalisation and the surrounding residual block are outside this operator count. A kernel that multiplies every stored tap, including the padded edge taps, would instead perform 99,840 band multiply-accumulates.

Against the specified dense baseline, HConvGLU stores about 20.9 times fewer parameters and uses about 21.7 times fewer band multiply-accumulates. Across 28 mixers the stored parameter counts are 2,809,856 versus 58,720,256. Neither ratio is an end-to-end latency ratio or an accuracy comparison. Memory access, transforms, activation functions and hardware utilisation determine the actual runtime.

Where it sits in Mirage

Five stride-two stem stages reduce the 512-pixel canvas to a $16 \times 16$ grid. The encoder then has 16 convolutional body blocks and 12 attention blocks. Each contains one HConvGLU at width 512.

In a body block, a gated depthwise convolution first mixes neighbouring spatial cells. A normalised feature vector enters HConvGLU, and global response normalisation acts on its output before the residual addition. The attention blocks instead follow their attention residual with normalisation, HConvGLU and a residual addition, without that response-normalisation wrapper. Valid-image masks keep padded cells out of the feature computation.

HConvGLU itself works on one feature vector at a time. Attention, spatial convolutions and the surrounding response normalisation supply the interactions between locations. Four register tokens participate in the attention stack and are removed from the image grid before the downstream heads use it. The Mirage architecture diagram and maths show these surrounding operations in full.

The inference implementation

The CPU engine processes small tiles of image cells. It computes the gate and value bands together to reuse the transformed input, evaluates the output band, and applies the second butterfly. It never constructs a dense $512 \times 512$ matrix for the band operation.

The HConvGLU bands and gate bias are stored in FP16 in the released archive and expanded to FP32 by the CPU loader. Mirage's 2-bit designation describes its main quantised matrices, not every tensor. A file-size estimate or a port needs to preserve this distinction.

The engine also folds the transform scales into the bands. Let $S=\sqrt n\,W$ be the unnormalised butterfly. It loads

and evaluates

The bias is not rescaled. This equals the normalised expression above. A port using normalised butterflies should use the original bands, while one using the folded weights should use the unnormalised butterfly. Applying both scales would change the operator.

HConvGLU limits the learned channel map to make the encoder smaller. Structured visuo-semantic perception imposes structure at another level, by reducing the language decoder's responsibilities. Both choices are explicit restrictions on the model. Their practical value is whether the resulting model performs the required perception tasks within the device's storage and compute budget. The Mirage release puts the complete system and its outputs in context.