Part III · Beyond SSMs Week 11 Published hyena_lineage.py test_hyena_lineage.py Notebook

Hyena, Long Convolution, and Linear Attention

Two parallel attempts to escape attention's O(L²): Hyena (eclipsed by mechanistic evaluation on associative recall) and the linear-attention lineage (which became the production frontier). The contrast sets up the W12 Unification Quadruple.

On this page

The Hyena operator
FFTConv: the O(Llog⁡L)\bigO{L \log L}O(LlogL) primitive
Why Hyena was eclipsed
Katharopoulos: transformers are RNNs
RetNet: retention as decay-weighted linear attention
GLA: hardware-efficient gated linear attention
HGRN2: outer-product state expansion
The UIA synthesis
[DYN] Stability of the matrix-valued state
What Week 11 buys you

Week 11 Overview

Goal: Read Hyena as the historical alternative to SSM recurrence, and trace the linear-attention lineage from Katharopoulos through GLA and HGRN2 as the load-bearing prequel to the W12 Unification Quadruple and the W13 production frontier.

Budget: 6–8 hrs (split: ~3 hrs Hyena + ~3–4 hrs lineage + 1 hr [DYN] stability aside).

Exercises: 2 core (Hyena FFTConv primitive in experiments/jax/week11/hyena_lineage.py; linear-attention recurrence by hand) + 1 [DYN] (Lyapunov computation on a matrix-valued state recurrence) + 1 optional benchmark.

Key insight: Hyena and the linear-attention lineage were two parallel attempts to escape attention’s $\bigO{L^2}$ . One was eclipsed by mechanistic evaluation on associative recall; the other generalised into the production frontier shipped in Qwen 3.5 / 3.6 and Kimi Linear. Reading them side-by-side is what makes the W12 Unification Quadruple inevitable rather than surprising.

Current production status (May 2026)

Hyena: eclipsed. No production model ships pure Hyena as of May 2026. The mechanistic-evaluation paper Anonymous (2025) showed Hyena fails associative recall in a way the original perplexity-only benchmarks did not surface. Production replaced Hyena with linear-attention variants once Zoology-style evaluation became standard.

Linear-attention lineage: the production frontier. The recurrence pattern that begins with Katharopoulos 2020 is the same recurrence pattern shipped in:

GLA-style chunkwise scans in Flash-Linear-Attention;
Gated DeltaNet ([?ch:week13]) in Qwen 3.5 / 3.6 and Kimi Linear’s KDA kernel;
mLSTM ([?w14:sec:mlstm]) in xLSTM 7B.

Two adjacent ideas, two opposite fates. The pedagogical payoff of W11 is that the contrast tells you why certain operator shapes survived.

The Hyena operator

Hyena Poli et al. (2023) replaces self-attention with an order- $N$ recurrence built from two operations: an implicit long causal convolution against a learned kernel, and an element-wise multiplicative gate. The order- $N$ operator interleaves these $2N + 1$ times:

\mathrm{Hyena}_N(u) = \bigl(\,v_N \elemwise h_N * \bigl( \cdots \bigl(v_2 \elemwise h_2 * (v_1 \elemwise h_1 * x)\bigr) \cdots \bigr)\bigr),

where $h_n$ are the learned long-conv kernels of length $L$ and the $v_n$ are linear projections of the input $u$ that supply content-dependent gating. Setting $N = 1$ collapses Hyena to a single gated long convolution; larger $N$ stacks more interleavings.

The architectural claim is that long causal convolutions plus multiplicative gating recover most of attention’s expressivity without paying the $\bigO{L^2}$ cost. The reasoning is the combination of two facts:

A causal convolution of length $L$ can be computed in $\bigO{L \log L}$ via FFT (next section).
Multiplicative gating supplies the content-dependence that a pure linear filter lacks — but without the per-token input-dependent convolution that Mamba’s selective scan requires.

The first fact is the operator-level performance argument; it works. The second fact is where the trouble starts: content-dependence-through-gating is a weaker form of selectivity than input-dependent dynamics, and the difference matters on tasks that require precise retrieval. The “Why Hyena Was Eclipsed” section below unpacks the failure mode.

FFTConv: the $\bigO{L \log L}$ primitive

The foundational primitive at the bottom of the Hyena operator is the FFT-based causal convolution. Given input $u \in \R^{B \times L \times D}$ and a per-channel filter $h \in \R^{D \times L}$ , compute

y_{b, t, d} = \sum_{s = 0}^{t} h_{d, t - s}\, u_{b, s, d} + \bm{b}_d \cdot u_{b, t, d},

where the trailing feedthrough $\bm{b} \in \R^D$ is the $\delta$ -tap that makes the operator a strict generalisation of a direct skip connection. The implementation pads $h$ and $u$ to length $2L$ along the sequence axis, takes the FFT, multiplies pointwise, inverse-transforms, and truncates to $L$ . The doubling is what makes the convolution causal under the cyclic FFT; without it, late entries wrap around to early positions.

Reuse from the S4 convolutional view

The FFT-conv machinery here is the same operator that the S4 convolutional view of [?ch:week04] computes. The difference is what gets plugged into $h$ :

S4: $h_t = \outputmat \discA^t \inputmat$ , the impulse response of a structured continuous-time ODE. The kernel is materialised from $(\statemat, \inputmat, \outputmat)$ .
Hyena: $h_t$ is parameterised directly by an implicit filter network (typically a small MLP indexed by $t / L$ ).

The FFT machinery does not care about the source. What it cares about is that the kernel has a closed-form expression at every position, which is true for both. Mamba’s selective scan ([?ch:week07]) trades the FFT’s $\bigO{L \log L}$ for an $\bigO{L}$ associative scan, but at the cost of fixing the kernel to a structured semiseparable matrix.

The implementation lives in hyena_lineage.py as fftconv(u, h, bias) with explicit dimension semantics. The test suite in test_hyena_lineage.py verifies the $\bigO{L \log L}$ output matches the explicit-Toeplitz $\bigO{L^2}$ reference to machine precision on random inputs across multiple shapes. The Toeplitz reference is what makes the chapter’s first exercise tractable as a correctness check rather than a benchmark.

Why Hyena was eclipsed

The Hyena paper reported near-Transformer perplexity at long context with sub-quadratic compute. The architecture nevertheless does not ship in production as of May 2026. Mechanistic evaluation explains why.

Why Hyena was eclipsed

Mechanistic evaluation Anonymous (2025) shows Hyena fails the associative-recall benchmark at sequence lengths where attention and DeltaNet succeed. The original “near-Transformer perplexity” claims rested on benchmarks that did not stress associative recall: language modelling perplexity averages over positions in a way that hides position-specific retrieval failures.

The mechanism: Hyena’s content-dependence comes only from the multiplicative gates $v_n \elemwise (\,\cdot\,)$ . The long convolution kernel $h_n$ is position-dependent but not input-dependent. To retrieve $v_j$ given $k_j$ at position $i > j$ , the model must store the entire $(k, v)$ association somewhere; a position-dependent filter cannot do this on its own. Multiplicative gates pass content through but do not let the kernel reshape itself based on what it has seen.

This is the same gap that Mamba’s selective scan was designed to close: input-dependent dynamics is strictly stronger than gate-on-output-of-fixed-dynamics. DeltaNet ([?ch:week12]) solves the retrieval problem differently — by maintaining an explicit associative-memory matrix $\bm{S}_t$ updated by an input-dependent rank-one rule. Either route works; the pure long-convolution-plus-gating route does not.

The lesson is not that long convolutions are useless. The FFT-conv primitive remains alive in S4’s convolutional view ([?ch:week04]) and in any operator that needs an $\bigO{L \log L}$ filter without the kernel-shape constraints of a structured scan. The lesson is that long convolutions plus multiplicative gating, treated as a complete architecture, lose to input-dependent recurrences on retrieval.

Katharopoulos: transformers are RNNs

The linear-attention lineage starts with Katharopoulos et al. 2020 Katharopoulos et al. (2020) . The contribution is a single re-derivation. Standard softmax attention is

\bm{o}_t = \sum_{s \le t} \frac{\exp(\langle \bm{q}_t, \bm{k}_s \rangle)}{\sum_{s' \le t} \exp(\langle \bm{q}_t, \bm{k}_{s'} \rangle)} \bm{v}_s.

Replace the kernel $\exp(\langle \cdot, \cdot \rangle)$ with a factorised feature map $\phi(\cdot)$ such that $\phi(q)^\top \phi(k) \approx \exp(\langle q, k \rangle)$ (or any other positive kernel one cares about). The numerator and denominator become

\bm{o}_t = \frac{\phi(\bm{q}_t)^\top \sum_{s \le t}\phi(\bm{k}_s) \bm{v}_s^\top}{\phi(\bm{q}_t)^\top \sum_{s \le t}\phi(\bm{k}_s)} = \frac{\phi(\bm{q}_t)^\top \bm{S}_t}{\phi(\bm{q}_t)^\top \bm{n}_t},

where $\bm{S}_t \in \R^{d \times d}$ and $\bm{n}_t \in \R^d$ accumulate by simple addition:

\bm{S}_t = \bm{S}_{t - 1} + \phi(\bm{k}_t)\bm{v}_t^\top, \qquad \bm{n}_t = \bm{n}_{t - 1} + \phi(\bm{k}_t).

This is the rewrite that makes softmax attention a recurrent neural network: $\bm{S}_t$ and $\bm{n}_t$ are $\bigO{1}$ -per-step state. Every successor architecture covered in this chapter (RetNet, GLA, HGRN2) and every architecture covered in the next three chapters (DeltaNet, Gated DeltaNet, mLSTM) starts from this recurrence and modifies it.

The cost of the rewrite is a quality regression: the feature map $\phi$ is an approximation to the exponential kernel, and the approximation is good only in the high-dimensional regime where softmax attention is doing relatively little reweighting. At “ordinary” depth and width, pure linear attention underperforms softmax attention on retrieval-heavy tasks. The successor lineage spends a decade closing this gap.

RetNet: retention as decay-weighted linear attention

RetNet Sun et al. (2023) introduces the first piece of the gap-closing: an exponential decay multiplier on the running state. The recurrence becomes

\bm{S}_t = \gamma\, \bm{S}_{t - 1} + \bm{k}_t \bm{v}_t^\top, \qquad \gamma \in (0, 1),

with $\gamma$ a fixed (not learned) per-head decay constant. The “retention” name is from the interpretation: $\gamma^{t - s}$ weights the contribution of position $s$ to position $t$ , so older positions are exponentially down-weighted. Mathematically, retention is the impulse response of a discrete-time first-order LTI system applied to the outer-product stream $\bm{k}_t \bm{v}_t^\top$ .

Where does RetNet sit in the W12 Unification Quadruple ([?w12:sec:unification])? Under the FWP lens Anonymous (2025) , RetNet is a fast-weight programmer with a fixed forgetting rate. Under the SSD lens Anonymous (2025) , the per-head scalar $\gamma$ is exactly the scalar SSM state-space parameter, so RetNet is a scalar-state SSM with the masked-attention reading. RetNet was the first architecture in this lineage to make both unifications visible at the same time; subsequent work (GLA, DeltaNet, mLSTM) replaces $\gamma$ with a data-dependent gate but keeps the structural shape.

RetNet was production-relevant briefly in 2023–2024 and is no longer the production endpoint — not because retention is wrong, but because fixed retention is too rigid. GLA fixes this.

GLA: hardware-efficient gated linear attention

GLA Yang et al. (2023) replaces RetNet’s scalar $\gamma$ with a data-dependent matrix gate. The recurrence is

\bm{S}_t = \diag(\bm{g}_t)\, \bm{S}_{t - 1} + \bm{k}_t \bm{v}_t^\top, \qquad \bm{g}_t \in (0, 1)^d,

where $\bm{g}_t$ is a per-channel forgetting gate computed from the current input. This is the first architecture in the lineage where the recurrence Jacobian $\diag(\bm{g}_t)$ is genuinely input-dependent — exactly the property whose absence cost Hyena the associative-recall benchmark.

The hardware-efficient contribution is the chunkwise scan algorithm. The GLA paper’s main practical claim is that the matrix state can be updated in chunks of length $L_c$ using a tiled matmul-plus-scan algorithm that maps cleanly onto GPU tensor cores. The structural form of the chunkwise scan is the same algorithm that Gated DeltaNet ([?ch:week13]) uses for its matrix-valued state; reading the GLA chunkwise code in Flash-Linear-Attention is how one earns the right to read the Gated DeltaNet code, not the other way around.

GLA is the direct precursor to the production architecture. The operator shape of the GLA recurrence is the operator shape shipped in Qwen 3.5 / 3.6 (under the Gated DeltaNet name, with the rank-one erase term added) and in Kimi Linear (under the KDA name, with per-channel data-dependent gating refinements).

HGRN2: outer-product state expansion

HGRN2 Qin et al. (2024) arrives at the same matrix-valued state shape from a different direction. The contribution is to start with a gated linear RNN (HGRN1) and expand the scalar cell state into a matrix via an outer-product trick:

\bm{S}_t = \diag(\bm{f}_t)\, \bm{S}_{t - 1} + \bm{i}_t\, \bm{v}_t \bm{k}_t^\top,

with forget and input gates $(\bm{f}_t, \bm{i}_t)$ derived from the input. Compare to mLSTM in W14: the structural shape is identical up to relabeling. HGRN2 and mLSTM were developed independently by different groups and converged on the same operator class — which is exactly the convergence the W12 Unification Quadruple Anonymous (2025) formalises.

The pedagogical role of HGRN2 in this chapter is to make the convergence visible. By the time the reader reaches W14, the matrix-memory recurrence will be familiar; HGRN2 is one of the “different groups arriving at the same place” data points that sets up the Unification Quadruple’s claim that these architectures are all viewing the same operator through different lenses.

The UIA synthesis

The Unified Implicit Attention paper Anonymous (2024) , ICLR 2025, proves the convergence formally. The result: Mamba, RWKV, and GLA all compute an implicit-attention operator of the same shape, differing only in parameterisation choices. RetNet, HGRN2, and mLSTM are corollaries of the same construction.

This is the FWP lens of the W12 Unification Quadruple ([?w12:sec:unification]) stated in operator-theoretic form. A reader who absorbs UIA at the end of W11 can read W12’s four-lens treatment as re-derivations of a single fact under different notations, rather than as four independent observations that happen to agree.

`[DYN]` Stability of the matrix-valued state

This section is the chapter’s first explicitly [DYN] exercise. The setup is general: every recurrence in this chapter has the form

\bm{S}_t = J_t\, \bm{S}_{t - 1} + \bm{i}_t \bm{v}_t \bm{k}_t^\top,

where $J_t$ is a matrix that determines whether the state grows, contracts, or sits on the stability boundary as $t$ increases. The choice of $J_t$ is the load-bearing architectural decision, viewed dynamically:

Architecture	Jacobian $J_t$	Stability
Pure linear attention	$I$	marginal (linear growth)
RetNet	$\gamma I$ , $\gamma < 1$	exponential contraction (fixed)
GLA	$\diag(\bm{g}_t)$ , $g_{t,i} \in (0, 1)$	data-dependent contraction
DeltaNet	$I - \beta_t \bm{k}_t \bm{k}_t^\top$	contraction iff $\beta_t \norm{\bm{k}_t}^2 < 1$

The pure-linear-attention case ( $J_t = I$ ) is the diagnostic instructive one. The state grows linearly in $t$ because every contribution $\phi(\bm{k}_s) \bm{v}_s^\top$ is preserved forever with weight one. Numerically, $\norm{\bm{S}_t} \to \infty$ as $t \to \infty$ , and the normalised output $\phi(\bm{q}_t)^\top \bm{S}_t / (\phi(\bm{q}_t)^\top \bm{n}_t)$ saturates: every value gets averaged equally, so retrieval signal washes out. This is the same retrieval-failure mode that Hyena failed for a different reason, and it is exactly what gated linear attention (RetNet, GLA, DeltaNet) is designed to prevent.

The Lyapunov diagnostic. The infinite-time growth rate of $\norm{\prod_{s \le t} J_s}$ is the maximal Lyapunov exponent $\lyapexp_1$ :

\lyapexp_1 = \lim_{T \to \infty} \frac{1}{T} \log \biggl\lVert \prod_{t = 1}^{T} J_t \biggr\rVert.

A positive $\lyapexp_1$ indicates a growing state (retrieval drift or exponential blow-up). A negative $\lyapexp_1$ indicates a contracting state (stable retrieval, possibly over-forgetful). A $\lyapexp_1$ at exactly zero is the marginal regime that pure linear attention sits in.

The numerical implementation uses iterated QR decomposition on the running matrix product, exactly the algorithm exercised in W01. Apply it to a trained GLA layer over real input sequences and the output is a diagnostic of whether the learned gate distribution keeps the state in the contracting regime — a generalisation of the spectral-radius monitoring instinct from classical numerical analysis.

Connection to implicit vs. explicit integration. The Longhorn lens of the W12 Unification Quadruple ([?w12:sec:unification]) views the recurrence as discrete integration of an online-learning ODE. In that frame, pure linear attention is forward-Euler integration with a unit step size and no decay — known to be unstable on stiff problems. GLA’s $\diag(\bm{g}_t)$ multiplier is the implicit-Euler-like correction: the gate acts as a per-channel step-size controller that pulls the integration back into the stability region. The forward-Euler-vs-implicit-Euler distinction from a graduate numerical-analysis course is exactly the right mental model for why retention / gating exists.

The Week 18 chapter, when authored, will apply the QR-iteration Lyapunov computation to trained Mamba models and compare the empirical spectrum across architectures. This W11 section establishes the framework; W18 turns it into a diagnostic.

What Week 11 buys you

Closing summary

After this chapter you can:

Implement the FFT-conv primitive from scratch and explain why the $2L$ -pad is what makes the cyclic FFT produce a causal linear convolution. The module is shared with S4’s convolutional view ([?ch:week04]).
Explain why Hyena was eclipsed without dismissing the long-convolution idea — the gap is content-dependence, not operator-level performance, and the gap is what input-dependent recurrences fix.
Locate every architecture from Katharopoulos to mLSTM in a single recurrence template $\bm{S}_t = J_t \bm{S}_{t - 1} + \bm{i}_t \bm{v}_t \bm{k}_t^\top$ and explain the production-vs-research outcome of each architectural choice for $J_t$ in dynamical-systems terms.
Apply the Lyapunov diagnostic to any of these recurrences. The diagnostic generalises immediately to trained models when the Week 18 chapter is authored.

The W12 Unification Quadruple ([?w12:sec:unification]) is the formal synthesis of what this chapter establishes informally: four different starting points (UIA, FWP, SSD, Longhorn) converge on the same operator class, and the linear-attention lineage is the empirical sequence in which the convergence was discovered. W13 ([?ch:week13]) then ships the conclusion in production.