Empirical Adjoint Learning · a five-level field game
Learn the way back by watching the way forward.
A network improves by sending blame backward from its mistakes. Backprop does it by running the forward wiring in reverse. SEA plays by stricter rules. Each layer watches what goes in and what comes out, fits a map between the two, and sends errors back through that fitted map. Five levels, 320 XP.
Level 1 · The Rulebook
What counts as learning without backprop?
The project's charter draws a hard line. Errors may not travel from the top layer to layer 1 by chaining gradients, and the return path may not reuse the exact forward weights. Sort each move.
Level 2 · Survey a Layer
Guess the map before SEA does
This layer takes a 2-D input x and emits ReLU(Wx + b). Drag the two cobalt arrows to build the linear map that best predicts its outputs from its inputs. Reach 90% of the best possible fit without help to earn the badge. The best fit is exactly what SEA computes from its covariances.
Inputs the layer sees
Outputs · actual vs your map's prediction
Each arrow shows where your map sends one input direction. Arrow keys fine-tune with the sliders below.
R² 0.00 · best possible 0.00
Level 3 · The Race
Four return paths, one spiral
Four copies of the same small network (2 → 24 → 24 → 2, ReLU, Adam) learn to separate two interleaved spirals. They start from the same weights and see the same batches. Only the path that carries errors back differs. Aim is the angle between the error a racer delivers to layer 1 and the exact backprop error on the same batch: 0° is perfect, 90° carries no information.
Level 4 · Tune the Instrument
Three knobs and a probe switch
Run SEA with your own settings on a fixed seed for 3,000 steps. The first run also records the paper's defaults on the same seed as your baseline. Beat it with any other setting to earn Tuner.
How long the running covariances remember. Higher is smoother but lags behind weights that keep changing.
Adds λI before inverting. Too small and noisy directions blow up. Too large and the map flattens toward an unwhitened copy of the cross-covariance.
Solve the map every k steps and reuse it in between. Caching is how the author's sea_fast gets close to backprop's cost.
Lab notebook
| Seed | ρ | λ | k | Probes | Accuracy | Aim |
|---|---|---|---|---|---|---|
| Your trials appear here. Accuracy is measured on 600 test points after 3,000 steps. Aim is the mean over the last 500 steps. | ||||||
One seed is noisy. A setting that wins once may lose on the next seed, so try a new seed before trusting a winner.
Level 5 · The Transformer Twist
Where does a token's error go?
In the language-model port, every residual block is one module. Each block's map is fitted on all tokens of the batch at once, and the error handed to the block below travels through that map one token at a time. Pick a prediction at the top, then compare the two return paths.
- Inside a block, attention's own weights still get a local gradient that uses every earlier position. That part is allowed.
- The readout map regresses 50,257 logits on the final hidden state. Logits are an exact linear function of that state, so the fitted map approximates the output weights, shrunk by the ridge in low-variance directions.
- The run in progress on 26 Sep 2026 uses 2 blocks, width 128, context 128 and 2,097,152 planned steps.
Debrief
Badges and field notes
Badge shelf
Tap again to clear your XP and badges.
What the evidence says so far
- At four hidden layers, orthogonal calibration beat data-calibrated SEA by 0.559 percentage points on Fashion-MNIST and 0.779 on MNIST, over 20 paired seeds with 95% intervals excluding zero.
- Against backprop, that calibrated arm was 0.396 points behind on Fashion-MNIST and 0.007 ahead on MNIST. The paper says the MNIST figure does not establish equivalence.
- Initialization plus training cost 1.28 to 1.44 times backprop. With cached maps (k = 10), sea_fast ran at 1.11 times backprop per step on the author's GPU at depth 2, width 256.
- The level 3 race uses Gaussian probes (the paper's SEA-noise). The replication numbers above use centered orthogonal probes.
- The paper's own limits: internal-holdout experiments that do not establish algorithmic novelty, general superiority or large-scale applicability.