Recirculation

1.1Once upon a time, models had memory#

Before transformers, language models processed text one word at a time, keeping track of what they had seen so far in an internal memory called their state. These models, known as recurrent neural networks (RNNs), basically apply a single function recursively:

z_t = f(z_{t-1}, x_t)

where x_t is the current input and z_t is the model's state.

While the idea of maintain a recursively updating state is intuitive and simple, the catch is that in practice, an RNN's state is not perfect and often forgets early parts of long sequences. Moreover, RNNs are slow. Step t cannot begin until step t-1 has finished, so processing a thousand-token sequence requires a thousand sequential steps. That can only be so fast.

1.2Transformers: everything, everywhere, all at once#

The transformer's central trick is to throw the sequential loop away entirely.

Instead of a state that gets passed along, every token talks to every previous token through attention. No token needs to wait for another. This means that given a fixed sequence, every position in a layer can be computed in one parallel sweep instead of going through the sequence one token at a time.

This single idea is a large reason why such effective language models exist today and why it is relatively economical to train these models on internet-scale data.

Unfortunately, there is no such thing as a free lunch.

1.3The cost of parallelism#

A critical fact is that state updates can only happen in the direction of flow of information, i.e., along the available directions of computation.

In an RNN, information flows along two axes: up through the network and forward through time. Importantly, the time axis allows states to undergo updates and flow back to where they originated from. In other words, an RNN layer always has access to an updated, contextualized state (at least in theory). Since state updates happen along the time axis, the only thing that limits the number of updates is how long the time axis can be which is simply the number of tokens the model sees/generates.

TLDR; an RNN's internal memory can be updated indefinitely.

However, in a transformer, there is no computational path along the time axis! Information flows only along the network axis. Activation at layer 8 can be read by layers 9, 10, 11. It can never be read by layer 3, at any token position. So, the number of times a state can be revised is capped by the maximum length of the network axis, i.e., the number of layers in the model.

This is the true cost of parallelism: the architecture inherently bounds the model's budget for state revision, one that sequence length simply cannot overcome.

1.4Fred goes to the bank#

The authors provide an example of this failure (produced by Gemini 2.5 Flash, and motivated by Lepori et al. (2025)):

User: Fred took the day off work and pulled out his fishing pole. He drove to the bank. When he reaches the bank, should he wear boots or flip flops?

---

Model: While flip flops may be more comfortable for a casual day, they are not the best choice for fishing, especially at a river bank. Here's why: ...

---

User: Is it likely that Fred will find an ATM at this bank?

---

Model: It is highly likely that Fred will find an ATM at the bank. Most banks, especially those located near bodies of water popular for fishing, have ATMs on-site for customer convenience.

The interesting thing is that even though the model itself recognized that Fred was going to a river bank in its first response, it still had no problem installing an ATM on the riverside. This is called a contextualization error, and it is not a knowledge issue but a state tracking problem. Having committed to a single contextual interpretation of bank, the model failed to hold on to that commitment and consult it later.

1.5Where the meaning of a word actually lives#

Why did that happen? Lepori et al. give us an extremely interesting answer, and it forms the crux of the entire paper.

Using Patchscopes, they traced the representation of a polysemous word like bank as it it processed through the network:

• In shallow layers, the embedding of bank is a blend of both meanings. The model is yet to disambiguate the context-appropriate meaning since it hasn't fully processed the surrounding context yet.
• In deep layers, once the model has read the given context, it is able to select the appropriate interpretation of bank by nudging its embedding decisively toward river edge.

The key insight here is that the model is fully capable of landing at the correct interpretation. It just forms it late, high up in its stack of layers.

Notice what happens when the next question is processed. The processing of tokens naturally starts from the first layer, which is completely blind to the top (the first layer never got to see its updated state!). Due to this, the early layers of the new question have access only to the ambiguous bank, which carries a strong statistical association with ATM. If the model settles on an answer before its deeper layers are able to re-resolve the ambiguity, it answers yes.

Lepori et al. call this a race: the model's response generation can outpace the model's internal semantic convergence. The right answer is available in the architecture. Unfortunately, it arrives only after an inaccurate decision has already been made.

1.6Looped transformers: taller, not longer#

If the problem is not enough sequential revision, the obvious fix is more depth. That is what a looped transformer does: instead of stacking L distinct blocks, it loops over a shared set of blocks several times over, getting the effect of a deeper network for free in terms of parameters.

In practice, while looping is fairly useful, there is a notable drawback in this architecture: it recurs only along the model's network axis. If we unroll this architecture, we can see that that this is not a loop at all! Instead, it is simply a taller feedforward transformer with shared weights.

Consequently, just as in a regular transformer, a state update z(t+1) = f(z(t), x(t)) causes the new state z(t+1) to sit one layer higher than z(t). Needless to say, the number of state updates is still capped by the number of layers in the architecture.

Can this cap ever be removed?

2.1The Crux#

Recirculation involves a very simple yet powerful idea: recur not only along the network but also along the time axis.

This simple idea is a game changer for tracking arbitrary states since the updated state can finally be seen by its original layer and can reside in it practically indefinitely, bounded only by the number of tokens in the sequence.

After each step, recirculation leaks a little bit of activation from a deep layer down to a shallow layer. It can run on off-the-shelf models without requiring any additional parameters or training.

2.2Do the Source and Destination Layers Speak the Same Language?#

The idea hinges on a source layer's activations being leaked to a destination layer. However, how do we know that these layers, with their different weights and activation statistics, can even understand each other?

The answer lies in the residual stream. A transformer's residual stream acts as a shared blackboard that all layers read from and write to, adding to the stream each time they compute something.

Let's say there is a single direction corresponding to the fact that given the current context, the word bank refers to river bank and not the financial institution. Since the residual stream is updated via addition (a commutative operation), it doesn't matter whether layer 4 writes to this direction or layer 11, both would have the same effect on the output distribution.

2.3What it Costs#

Unfortunately, recirculation's performance improvements come at a notable cost. Since the state from step t must exist before step t+1 runs, the prompt has to be processed autoregressively, token by token. In other words, it cannot be parallelized even when all input tokens are known. For long contexts, that is quite slow.

3.1Models#

The main results use the Gemma3 family at 1B, 4B and 12B, in both PT (pretrained) and IT (instruction-tuned) variants. Cross-family robustness is checked on Ministral3 3B, Qwen3 1.7B, Pythia 1B and Phi2 2.7B, plus the older Gemma2 and newer Gemma4 generations.

3.2Evaluations#

1. Perplexity on ten language-modelling datasets over 1024-token windows.
2. Instruction following on a purpose-built executive-function task.
3. Contextualization on the Racing Thoughts dataset.
4. Single-token and multiple-choice benchmarks
5. GSM8k with zero-shot chain-of-thought, at pass@1 and pass@128.

4.1Where does the model want the loop?#

The authors first determine the optimal values of s and d, the indices over the source and destination layers, respectively, for the Gemma3 models.

With larger values of \alpha, source-destination pairs where recirculation works lead to larger gains while the number of pairs which hurt perplexity also increases. It is, thus, important to find a value of \alpha that manages this tradeoff well.

Similar sweeps over Gemma3's 4B and 12B variants yields the following pairs for each model variant:

Averaged over arXiv, PG19 and C4 at \alpha = 0.10, the best pair reduces perplexity by 4.72%.

4.2A Bug?!#

The arXiv version of this paper contains an unusual note from the first author warning readers about a bug in the perplexity experiments!

The team spent months suspicious of a 30% perplexity reduction on an off-the-shelf model and three authors independently reimplemented the algorithm from scratch. The fatal flaw was in the data loader which was shared among the authors.

The loader placed a beginning-of-sequence (BOS) token at the start of each document, not at the start of each 1024-token window. Most model families do not care. As luck would have it, Gemma expects a BOS at the start of every window. Without it, its baseline perplexity is inflated which makes recirculation's improvement look far larger than it is.

What this does and does not affect:

• Unaffected: every non-perplexity evaluation. They are all single-window tests that begin with a BOS. External groups have independently replicated and extended the accuracy results.
• Affected: the magnitudes in the table below, and the roughly 5% figure quoted for Gemma in the cross-family comparison.
• Still true: Gemma models show consistent perplexity reductions once the confound is removed but on the same order of magnitude as the other model families, which is to say smaller than reported here.
• Still true: the qualitative pattern of the sweeps. That is the part that matters most, since the purpose of the sweeps is to locate the model's affordances, not to post a headline number.

Read the numbers in the next subsection as the shape of a result rather than its size. I will remove this section and update the numbers on this page once the authors update their paper.

4.3Perplexity#

With source and destination fixed from the sweep and \alpha = 0.15, evaluated on held-out splits:

Nine of the ten datasets improve at every model scale. Lambada is the exception and the authors claim that it's because it consists of very short sequences and carries tokenization artifacts. Recirculation is a mechanism for accumulating state over time, so a dataset with almost no time over which to accumulate is precisely where it doesn't have anything to offer.

4.4Does It Work Outside Gemma?#

Ministral3 3B, Qwen3 1.7B, Pythia 1B, Phi2 2.7B, Gemma2 and Gemma4, all show the same qualitative signature: a robust region of the source–destination heatmap where perplexity falls, concentrated in the middle of the architecture.

The magnitudes differ sharply: about 5% for Gemma versus under 0.5% for the others. However, as the BOS note explains, that gap is largely the artifact rather than a real property. Once corrected, the families land in the same range.

4.5Ruling Out the Boring Explanations#

Since perplexity responds to anything that sharpens or flattens the output distribution, the authors questioned if the improvements were actually a consequence of recirculation or if perhaps some other factor was playing a bigger role here.

Is it just temperature? Adjusting the softmax temperature of the base Gemma3 1B to 1.2 reduces perplexity by 8.48% all on its own. Recirculation alone gives 14.21%. Combining them gives 19.55%. If recirculation were secretly a temperature adjustment in disguise, stacking the two would be redundant and the combination would not clear either one by much. It does, so recirculation is doing something temperature scaling cannot do.

Is it just looping? The authors implement training-free looping under a matched protocol: for every layer pair (\ell_1, \ell_2) with \ell_2 > \ell_1, insert a second copy of layers \ell_1{+}1 \ldots \ell_2 immediately after \ell_2, and sweep. Two differences emerge:

1. For the Gemma3 family, training-free looping does not produce robust benefits for this family, while recirculation does.
2. Recirculation helps across all three model scales; looping a pretrained model helps only at the larger scales.

The heatmaps come out qualitatively different in shape, further supporting the idea that recirculation is not just looping in disguise.

4.6Does Recirculation Benefit All Tokens Equally?#

The authors try to determine which kinds of tokens benefit most from recirculation. They conduct this analysis from 3 separate lenses:

1. By lag: After recirculating the token at position t, the authors measure the change in log-likelihood at position t+k for k \in [1, 256] where k is known as that token's lag.
2. By position: Which token positions benefit from recirculation the most?
3. By part of speech: Do certain grammatical classes benefit more from recirculation?

4.7Generative Performance#

Apart from perplexity, the authors observe the effect of recirculation on different generative capabilities of the model:

4.7.1Instruction Following#

In order to gauge if models' instruction-following capabilites are enhanced, the authors prompt the models (Gemma3 4B & 12B) with text that included the following instructions:

Let's play a game. I will say two words. If the first word is a fruit, you say first. If the second word is a fruit, you say second.

These simple instructions serve as quite a useful test of executive function in children and neurological patients.

Using the hyperparameters that minimize perplexity on a tuning set, the models already show marked improvements in accuracy. The authors also test the upper bound of these improvements by performing a hyperparameter sweep meant specifically to overfit to this instruction-following task. Doing so increases performance even further.

4.7.2Contextualization:#

This paper started with a contextualization error and would be incomplete without an analysis on whether such errors were ultimately reduced or not.

The authors use Lepori et al.'s dataset to test this. Each sample contains a context, a cue, and a yes/no question, with zero or more distractor sentences wedged in between:

I am holding a fishing rod. The game's success led Sega to develop an extensive media franchise. I see a bank. Is it a financial institution?

alongside factual and gender-related variants:

Forget everything you know about geography. The capital of Egypt was just renamed from Cairo to Beirut. Is the capital city of Egypt Cairo?

The soldier is somebody's grandmother. Is the soldier a man?

Moreover, to reduce the model's probability of guessing the right answer, it must respond correctly to two variants of the same question for its responses to be considered correct.

As far as the results go, recirculation's effect on the three Gemma3 models is mixed:

• Gemma3 1B IT: two of the three question types improve clearly; the third is at chance for baseline and recirculation alike.
• Gemma3 4B IT: two of three improve reliably; the third is about the same.
• Gemma3 12B IT: two of three are noticeably worse with recirculation; the third is at ceiling for both.

While technically, this is still a win overall, in the authors' own words, its failure on 12B is disappointing.

4.7.3Multiple-Choice & Single-Token Response Tasks#

The authors utilize fairly standard benchmarks to test model accuracy (of Gemma3 4B PT) across various domains.

The accuracy increases for most tasks but not by notable margins (<1\% difference). Perhaps the single-token responses that these tasks score do not rely as much on state accumulation due to the absence of long context which is why the gains on these tasks are limited.

4.7.4GSM8k: Long-Context Reasoning#

In contrast to the single-token responses being generated in the previous experiment, GSM8k prompts the model to solve grade school math problems using zero-shot chain-of-thought. This allows for an evaluation of the effect of recirculation on a model's long-context reasoning capabilities.

4.8Adaptive recirculation#

Till now, recirculation has been analyzed in a completely training-free setting. This is a double-edged sword. While it allows the use of recirculation on off-the-shelf models with minimal interventions, it has the downside of requiring extensive grid searches to find good values for hyperparameters such as \alpha and \beta.

The natural extension? Learn \alpha and \beta! The authors call this idea adaptive recirculation and they analyze the following variants of the same.

As evident, it even edges out full fine-tuning of Gemma3 1B while leaving the model's weights completely untouched, thereby bypassing the risk of overfitting. Additionally, various variants of adaptive recirculation are tested on downstream benchmarks with each variant being tuned on a different dataset. When tuned on the large and well-curated MMLU test dataset, adaptive recirculation beats vanilla recirculation on each benchmark. However, its performance degrade when tuned on lower quality (MMLU train set) or smaller (ARC Easy/Challenge) datasets.

The authors are extremely careful about what they are claiming. Apart from the methodological novelty that this paper presents, there is a deeper philosophical argument set forward by the authors which is quite interesting.

The authors talk about the concept of an affordance in product design, i.e., a property of an object that tells you how it should be used. A door handle affords grasping and pulling. Its shape is the instruction.

The authors propose applying this concept to foundation model research. Instead of proposing potentially effective architectural modifications and then spending time and effort in testing their efficacy, maybe we can look inside a trained network and exploit its natural properties to discover certain useful inductive biases that could make subsequent training easier. The adaptive-recirculation results, where tuning two coefficient vectors surpassed full fine-tuning, are offered as evidence for exactly that hypothesis.

Prefill is serial. The most practically limiting item, and the one that would block deployment on long-context models today. A promising fix the authors have not yet explored: blockwise recirculation, processing K tokens in parallel and then recirculating those K together with the next K. The trade-off between K and performance is unstudied.

Hyperparameters are not yet universal. The best (s, d, \alpha, \beta) appears to be domain- and problem-dependent. While they transfer well across tasks, a task-universal setting or an automatic task-to-hyperparameter map is still missing, thus limiting practical applicability.

Dependence on model family. The qualitative pattern holds everywhere tested, but Gemma's Peri-LN design and training recipe may make it unusually receptive. Generalization to other architectures needs further investigation.

Normalization is empirical. Norm-matching the source makes recirculation behave; the authors found one scheme that works across Gemma sizes. Whether normalization is necessary, and what the right scheme is for other families, is open.

Only one iteration, and only one path. Every experiment recirculates each stack exactly once. Running r iterations is cheap on modern hardware and unexplored. Likewise, the authors pick a single source–destination pair but the 4B and 12B sweeps show distinct, disjoint regions where recirculation works, which hints that multiple simultaneous paths might carry state at different levels of abstraction and turn out to be complementary.

Adaptation has barely been touched. The adaptive results came from a small tuning set with no weight updates at all. Token-conditioned recirculation pathways and learned normalization schemes are next in line, well before anyone needs to reach for fine-tuning.

Looped transformers loop over a range of layers multiple times, obtaining a deeper architecture with no additional parameters. Looping can be deterministic or adaptive and genuinely increases expressivity. However, it recurs only in depth, so it does not guarantee indefinite state tracking. Some looping methods are designed and trained for inference-time scaling, others incorporate recurrence through pretraining or fine-tuning, and a few operate purely at inference time.

Recurrent transformers do express arbitrary state dynamics — a few token-by-token, most blockwise. State-space models are often described as state propagators, but many are no more expressive than an ordinary transformer, and none match a generic recurrent net. The closest relative to this work is T2MLR, which independently recognizes the value of feeding a deeper layer's representation from the previous token position into a shallower layer of the current one. It diverges in two ways that matter: T2MLR requires Backpropagation Through Time (BPTT) training or fine-tuning, and it parameterizes the recurrence pathway with a dedicated learnable gated fusion module. Recirculation needs neither, because it leans on the residual stream's intrinsic representation alignment instead.

Thinking models solve the depth problem in a different way by allowing a model to talk to itself via chain-of-thought. The model sends signals from deep in the transformer back to shallow layers through the token stream, in natural language or in latent space. The authors' framing is that chain-of-thought is the right tool for complex inference but the wrong tool for basic state tracking, which should not have to be spelled out in words to be remembered.

Activation steering modifies behaviour by intervening on latent representations, usually with a static, externally derived steering vector. Recirculation fits neatly into that frame with one twist: it is self-steering. The vector is not imported from the outside but is the model's own contextualized deep-layer activation fed to a shallower layer in the next time step.

State tracking theory underwrites the whole motivation. Transformers are bounded in serial capacity by depth, and a state representation becomes harder to use as it drifts upward into deeper layers. Merrill and Sabharwal (2025) prove that \log n layers are necessary and sufficient to recognize regular-language strings of length n and graph-connectivity problems on n vertices. However, that is a statement about what can be constructed, not about what can be learned. In practice, depth-limited models trained on specific finite-length problems find clever solutions to track states. Recirculation proposes that they should not have to.