Gabor Initialisation with Matched Filter Energy

A $3\times3$ spatial filter has just nine positions with which to respond to an edge or a change in texture. We can fill them with independent random numbers, as usual. We can also give some filters a pattern to start from, while leaving them free to learn something else.

In Mirage, we use a mixture. Half of the output filters in selected spatial convolutions keep their Gaussian weights. The other half start as Gabor patterns, with positive and negative regions arranged along a direction. Each replacement keeps the norm of the filter that was there before. We change the pattern while preserving its weight energy.

Gabor initialisation has a history in CNNs. For example, a 2023 study uses Gabor filters that remain free to change during learning. We'll build up our version from the spatial pattern, then look at what matching the norm means geometrically. That distinction matters because equal-sized weights can still respond very differently to an image.

Give the filter a direction

Imagine a set of alternating light and dark bands, strongest near the middle and fading toward the edges. A real Gabor pattern describes that shape by multiplying a sinusoid by a Gaussian envelope. To turn the pattern, we first rotate the coordinates,

$x'=x\cos\theta+y\sin\theta,\qquad y'=-x\sin\theta+y\cos\theta.$

Our sampled spatial template is

$g(x,y)=\exp\!\left[-\frac{x'^2+\gamma^2y'^2}{2\sigma^2}\right]\cos(2\pi f x'+\phi).$

$\theta$ turns the coordinate axes. The bands run along lines of constant $x'$, perpendicular to the direction in which the sinusoid varies. Increasing $f$ brings them closer together, while changing $\phi$ shifts their positions. The envelope width $\sigma$ and aspect ratio $\gamma$ control how much of that pattern fits around the centre.

Our $3\times3$ kernel samples this continuous shape at coordinates $-1,0,1$ on each axis. With only nine samples, you won't see a finely resolved wave. What remains is a small arrangement of positive and negative weights with some directional structure. The sampled kernel, including its truncation, is what the network actually receives.

The template generator draws independent parameters per output filter,

Here, frequency is measured in cycles per feature-grid pixel. Deeper in the encoder, one grid step covers a larger part of the photograph. Keep that change of scale in mind when comparing filters from different layers.

Remove the constant response

Sampling a Gabor on a small grid generally leaves a nonzero mean. We subtract the mean of the actual sampled kernel,

$\bar g=\frac1{hw}\sum_{x,y}g(x,y),\qquad g_0(x,y)=g(x,y)-\bar g.$

The positive and negative weights now sum to zero. On a constant patch, their contributions cancel, provided the filter lies away from the padding boundary. Across an edge, the two sides can receive different values and produce a response. This gives the initial filter a preference for spatial change.

For a convolution with $c_{\mathrm{in}}$ input channels, draw $a_c\sim\mathcal N(0,1)$ and form

$G_{xyc}=g_0(x,y)a_c,\qquad \widehat G=\frac{G}{\max(\lVert G\rVert_F,10^{-12})}.$

A depthwise filter has one input channel. For the first RGB convolution, we reuse one spatial pattern across the three colour channels and give each channel a random multiplier. The initial filter is separable into a spatial pattern and a colour direction. Once learning begins, every weight is free to move, so this structure can change.

Match the filter that was there

Let $W_o$ be one output filter from the original Gaussian initialisation. We first retain its norm,

$r_o=\left(\sum_{x,y,c}W_{xyco}^2\right)^{1/2}.$

For a seeded subset $S$ containing $\lfloor c_{\mathrm{out}}/2\rfloor$ output channels, the hybrid weights are

This keeps the untouched channels bitwise identical to their Gaussian counterparts. Each replaced filter retains its own original norm, up to floating-point rounding. We match the whole output filter across its spatial and input-channel entries, so a large original filter and a small one keep their respective scales.

Selection and template sampling use deterministic streams derived from the seed and the layer path. Separate streams keep the selected channel indices independent of the random draws used to construct the templates. A paired comparison can therefore reuse the original Gaussian tree without changing unrelated weights.

In the figure, switch between the original and replacement weights. The signs and spatial arrangement change, while the norm printed below each filter stays the same. Half the filters retain the original random directions, giving us a mixture of structured and unstructured starting points. The fraction is a design choice that can be tested against the all-Gaussian control.

Moving around a sphere of equal energy

Flatten a filter into a vector $w\in\mathbb R^d$, where $d=hwc_{\mathrm{in}}$. A depthwise $3\times3$ filter lives in nine dimensions; the first RGB filter lives in 27. All filters with norm $r_o$ lie on a sphere,

$\mathcal S_{r_o}=\{w\in\mathbb R^d\mid\lVert w\rVert_2=r_o\}.$

The replacement chooses another point on this sphere. Its distance from the origin stays fixed, while its direction changes. This is a direction in weight space. The angle $\theta$ earlier described orientation on the two-dimensional image grid.

Centring adds another geometric condition. For each input channel, the spatial weights sum to zero,

$\sum_{x,y}W'_{xyco}=0.$

Each of these linear constraints defines a hyperplane through the origin. The replacement lies where those hyperplanes meet the sphere, with the Gabor construction further restricting its spatial pattern. These conditions describe the starting weights only. We don't constrain later updates to stay on that sphere or preserve the Gabor shape.

Now place an input patch $z$ in the same vector space. The linear filter response is

$w^Tz=\lVert w\rVert_2\lVert z\rVert_2\cos\alpha,$

where $\alpha$ is the angle between the two nonzero vectors. A fixed-length filter can respond more strongly simply by aligning better with the patch. Matching weight energy therefore leaves room for exactly the directional change we're introducing.

What the image distribution changes

We can make that picture more precise over a collection of patches. If their covariance is $\Sigma$, the variance of a linear filter response is

$\operatorname{Var}(w^Tz)=w^T\Sigma w.$

If $\Sigma=s^2I$, equal weight norms imply equal response variance,

$\operatorname{Var}(w^Tz)=s^2\lVert w\rVert_2^2.$

In this special case, the patch distribution has equal variance in every direction, so all points on the sphere give the same response variance. Image patches and intermediate features are correlated. Their covariance can stretch some directions more than others, and $w^T\Sigma w$ changes as we move around the sphere. Equal weight norms therefore give us a controlled starting scale without guaranteeing equal activation distributions.

The constant patch we considered earlier is a simple example. The centred Gabor weights cancel, while a particular Gaussian draw usually leaves a nonzero response. Equal norms allow both behaviours.

Where the initialisation applies

The default scope covers the encoder's first spatial convolution, its depthwise stem convolutions and the depthwise convolutions in its body spatial mixers. Pointwise matrices, attention, detection and mask heads, the OCR reader and HConvGLU parameters keep their own initialisation.

This separates two kinds of structure in Mirage. Gabor patterns organise the initial spatial filters. HConvGLU restricts the channel-mixing operator throughout inference. A Gabor-initialised convolution can later become any convolution of the same shape. HConvGLU continues to use its fixed Walsh basis and learned bands after learning.

For someone deploying the model, this choice stays out of the way. The checkpoint contains ordinary convolution weights, and the runtime loads them as usual. There is no Gabor pattern to reconstruct and no extra inference operation. The kernel shapes and parameter counts stay the same.

Checking what we changed

We ran a small numerical check on a 32-output RGB kernel and a 64-channel depthwise kernel, both $3\times3$. Exactly 48 of the 96 output filters changed. The other 48 remained identical. The replaced filters retained their original norms within FP32 rounding and had zero spatial sum within numerical tolerance.

This checks that the implementation changes the directions and preserves the scales as intended. To learn whether that starting point improves recognition or convergence, we need completed comparisons that vary initialisation independently, use the same evaluation conditions and repeat across seeds. The numerical check answers the smaller question it was designed for.

You can follow the resulting image features into the instance mask head and see how the separate CTC reader handles text. The Mirage article brings these parts together, with a playground for trying them on your own images.