01

Why Nested Learning?

Nested Learning treats a model as several coupled optimization problems that run at different speeds or levels. Standard deep learning draws a hard line between architecture and optimization. Nested Learning instead treats layers, optimizer state, momentum, and attention caches as stateful learners with update rules.

The practical goal is plasticity without amnesia: components that need to adapt quickly can update frequently while knowledge-heavy components move on slower clocks. The update frequency, rather than the component name, determines its level.

02

Associative memory, update frequency, and levels

A module M can be viewed as an associative memory that learns a mapping from keys K to values V by minimizing an internal objective L̃(M(K), V). This view applies to linear layers, attention key-value stores, and optimizers that compress a stream of gradients.

Let one data-point update be one unit of time. A component A is faster than B when A updates more often, or when both update at the same frequency but B depends on A’s current state. Sorting components by that relationship produces levels: higher levels update less frequently.

03

Geometry of the deep optimizer

For a linear map y = Wx, Nested Learning introduces an inner L2 objective that regresses Wx toward the output gradient gy. One inner step gives W′ = W(I − αxxᵀ) + αgyxᵀ. Combined with the outer gradient step, the update can be written as Wt+1 = Wt(I − αxtxtᵀ) − η∇W L(Wt; xt).

The factor I − αxxᵀ is a right-side projection in input space. It reduces sensitivity most strongly along the current input direction and changes orthogonal directions least. For a mini-batch, xxᵀ becomes the batch Gram matrix XᵀX/B.

Comparison of linear-map outputs before and after the Nested Learning right-side projection for directions around the unit circle
The projection changes the map most strongly along the current input direction.
Projection update magnitude plotted against the angle between a direction and the current input
Update magnitude is largest parallel to the input and smallest in orthogonal directions.
Unit circle transformed into an ellipse by a batched Gram-matrix projection
For a batch, the Gram-matrix eigenvectors determine the shrink directions.

04

Continuum Memory Systems

A Continuum Memory System composes memories with different clocks. Each level accumulates gradients continuously but applies an update only when its period is due. Geometric periods such as 1, 8, 64, and 512 create a hierarchy from fast sequence-local adaptation to slower knowledge-heavy updates.

The schedule separates how often evidence is observed from how often each state is changed. That distinction makes the clock assignment explicit and testable.

Multi-timescale schedule with three Nested Learning levels updating at different periods
Only levels whose period is due apply an update; the others continue accumulating gradients.

05

Multi-clock training sequence

During each global training step, every block participates in the forward and backward pass. Gradients accumulate for each block, the trainer checks each period, and only due blocks apply their optimizer and projection updates. The remaining blocks keep their accumulated state for a later clock tick.

At a step where periods 1, 16, and 128 are all due, all three levels update together. At intermediate steps, only the faster levels move.

Nested Learning training sequence showing gradient accumulation, clock checks, and conditional optimizer updates
A global step accumulates evidence for every block and updates only the clocks that are due.

06

Runnable reference implementations

The reference repository contains four implementations. nested_learning_minimal.py demonstrates DeepL2GD, a Continuum Memory System, and sequential-task measurements. deep_l2gd_ema.py smooths Gram matrices with an exponential moving average. nl_conv.py applies a channel-covariance approximation to convolutional layers. nl_transformer_tiny.py wires a small transformer with separate clocks for attention and feed-forward paths.

  • Fast clock: query, key, and value projections
  • Intermediate clock: attention output projection
  • Slow clock: feed-forward network
  • Sequential tasks: measure both adaptation and retained accuracy
Transformer clock assignment with query, key, and value projections updating fastest and the feed-forward network slowest
A practical multi-clock transformer assigns faster periods to sequence-local paths and slower periods to knowledge-heavy paths.

07

Practical recipes and reproducibility

Start with geometric clock periods such as 1, 8, 64, and 512. Assign fast clocks to sequence-local paths and slower clocks to output projections and feed-forward blocks. Begin the deep-optimizer strength α between 10⁻⁴ and 10⁻³; larger values can underfit dominant directions. Use an exponential moving average when mini-batches are small or noisy, and per-batch Gram matrices when rapid adaptation matters more.

Run the minimal example to report Task A and Task B accuracy before and after sequential training. Run the transformer example on two synthetic bigram regimes to observe adaptation under distribution shift. The original code and experiment entry points remain available in the NestedLearning repository.

08

References

Google Research: Introducing Nested Learning · November 7, 2025.

Nested Learning: The Illusion of Deep Learning Architectures · Behrouz, Razaviyayn, Zhong, and Mirrokni · NeurIPS 2025.

Titans: Learning to Memorize at Test Time · Behrouz, Zhong, and Mirrokni.