Author(s): Abdul M.

Originally published on Towards AI.

A from-scratch DQN trained for 76 hours on a laptop CPU. In 15 ablation runs, the plainest mechanism mattered most, and two of the clever ones hurt.

I wanted an agent that learns from its own failures and never stops training. The setup was small on purpose: a side-scrolling runner game, a laptop with no GPU, and everything written from scratch on NumPy. That meant the game, the network, backpropagation, Adam and the reinforcement learning, with no PyTorch, no Gym and no pretrained weights. Then I let it run for days and measured everything.

The agent got good. At the harder tiers it beats a hand-written player that reads the level directly. But the most interesting results were the ones that went the wrong way. I built two mechanisms specifically to make it learn from failure, and taking either one away made it better. The mechanism that mattered most was the plainest one in the system. And over 226 million steps the network slowly lost its ability to learn, exactly as recent research predicts. Fixing that kept the network healthy, but only sometimes made the agent play better.

The code, the raw metrics and every number below are in endless-learner.

The setup

The game. A runner with low and tall blocks, gaps, overhangs to duck under, and blocks that slide towards you. From tier 5 on, these come in tight combinations. The agent has four actions: nothing, short jump, long jump, duck. A level is a pure function of a tier and a seed. Tiers have no top: speed, density and the obstacle mix keep rising until they hit limits set by the physics. Before any level is handed out, a depth-first search over jump timings checks that it can be finished. The search runs on the same physics code as the game, so no death is ever the level’s fault.

The learner. Double DQN (van Hasselt et al., 2016) with 3-step returns and a target network, on a 155–128–128–4 MLP. The forward pass, backward pass and Adam are written out by hand and checked against central differences, with a max relative error around 1e-8. It runs at a median of 870 to 1,180 environment steps per second on one core of an i7–1265U. That’s slow next to a GPU, but every gradient is visible.

Learning from failure. Two mechanisms on top of plain replay:

• Prioritised experience replay (Schaul et al., 2016) samples transitions in proportion to their TD error. Transitions just before a death are the surprising ones, so they get replayed more.
• A death bank. Every death saves the game state from 20 decisions earlier, and a quarter of all episodes restart from one of those states. The agent practises the exact situation that killed it, the idea behind Go-Explore’s “first return, then explore” (Ecoffet et al., 2021) and reverse curriculum generation (Florensa et al., 2017).

Never stopping. A curriculum promotes the agent when it passes 80% of fresh levels at its tier, and levels with high TD error are replayed more (prioritised level replay, Jiang et al., 2021). To avoid forgetting, 15% of episodes replay a tier it has already passed, and a reservoir sample of the whole history makes up a quarter of every training batch.

Measurement. Every 200k steps, the agent plays 20 held-out levels per tier with no exploration. Those seeds are never trained on. One eval round is noisy, so every number here is an average over a window of rounds.

Three things that broke first

Before any of the interesting results, three bugs in my thinking had to go. Each one cost me a day, and I think each one is a general lesson.

The agent couldn’t see gap edges. My first observation was nine rays fanning out from the agent. A fixed fan touches the ground at only four distances (0.8, 1.6, 3.1 and 8.2 units ahead), so the agent knew a gap existed but not where its edge was. In four runs of 1.2M steps, none got past tier 1, the first tier with gaps, and gaps killed the agent in 26 to 44% of its exams. Adding a terrain scan, the course sampled every half unit for 12 units ahead, fixed it: all four runs passed tier 1 within 230k to 340k steps. What the agent can observe sets the ceiling, and no amount of learning gets around it.

Exploration hid how good the policy was. With 3% random actions per decision, a timing-critical game kills you a lot. Exploring episodes passed about 40% of tier-0 levels while the same network, played greedily, passed 100% of the held-out ones. The curriculum was promoting on the wrong signal. Now 30% of fresh levels are played as greedy “exams”, and only exams count for promotion.

The policy changed under its own feet. This one surprised me most. With learning frozen, a snapshot of the network passed 100% of 100 random tier-0 levels. The same network playing while it trained passed about 55%. Each update flips the greedy action in some states, a well-known effect called policy churn. An episode lasts about a hundred updates, and here a jump one tick late is a death. The fix was to act with an exponential moving average of the weights (Polyak averaging, 0.5% per update) instead of the online weights. Over two seeds, that version reached frontier tier 4 on both, against tiers 2 and 4 for the online weights. Two seeds is a direction, not a measurement.

The climb and the plateau

With those fixed, the main run climbed fast: tiers 1 to 7 at 0.09M, 0.25M, 0.42M, 0.60M, 0.75M, 3.3M and 6.3M steps. That’s 2.4 active hours for all seven promotions. Then it spent the remaining 220M steps, 74 more hours, between tiers 5 and 7, and never held tier 8. One other run was promoted to tier 8 at 19.8M steps and demoted again at 34.2M.

For scale, I wrote a rule-based player that reads the level directly and times each jump to centre its arc over the next obstacle. It’s perfect up to tier 4, then passes 60% of tier-5 levels and 30% at tiers 6 and 7. The learned agent gets 52 to 65% at tier 6 and 49 to 63% at tier 7, depending on the run. At tiers 6 and 7 it beats the player with perfect information, which I didn’t expect.

The ablations

To find out what each piece was worth, I removed one at a time: five variants, three seeds each, 5M steps per run. The score is held-out success on tiers 0 to 5, averaged over the eval rounds from 4M to 5M steps.

variant mean range over 3 seeds
full config 0.76 0.74 to 0.79
no prioritised replay 0.88 0.86 to 0.92
no death-bank restarts 0.88 0.84 to 0.90
no curriculum (always tier 6) 0.89 0.83 to 0.91
no old-tier replay 0.21 0.14 to 0.29

Two findings stand out.

Replaying old tiers is what holds everything together. Without it, every seed reached tier 7, at 1.9M to 3.3M steps, sooner on average than any other variant. Meanwhile it forgot the easy tiers almost completely: averaged over the seeds, it passed only 5 to 23% of tier-0 to tier-3 levels, where the full config passes 61 to 96%. That is catastrophic forgetting in its plainest form (Kirkpatrick et al., 2017). I think a curriculum makes this worse by design, because it is a schedule for leaving old data behind.

The two failure mechanisms made the agent worse. So did the curriculum. Removing any one of the three beat the full config on every seed, and the seed ranges don’t overlap.

What makes this interesting to me is that both failure mechanisms did their local jobs well. In the main run the death bank restarted 323,444 episodes and fixed 23,117 failures, in a median of 2 attempts, without giving up on a single one. Transitions just before a death carried 5.5 times the TD error of routine ones (0.56 against 0.10), so prioritised replay was focusing exactly where it was meant to. Every local metric said “learning from failure, working”. The global metric said the agent was better off without it.

My guess is that each mechanism narrows what the agent trains on, and that narrowing costs more than the focus gains. Prioritised replay oversamples surprising transitions. The death bank spends a quarter of all episodes on fragments just before a death. The curriculum keeps the agent on one tier at a time. Each skew is small, but they stack. I haven’t tested that explanation directly, and these results are for a 5M-step budget only: with much longer training, the balance could change.

The general lesson I’d take is about metrics, not about replay. A mechanism can have excellent numbers of its own, like failures fixed or TD error concentrated where it should be, and still not move the number you care about. You only find out by taking it away.

226 million steps: the network loses plasticity

The second question was what happens to a network that never stops training. Dohare et al. (Nature, 2024) showed that networks trained on a changing task gradually lose the ability to learn. Units die, the representation collapses, and new things get learned worse than a fresh network would. A curriculum without an end is exactly that kind of setting.

It showed up clearly. I tracked three things:

• the share of hidden units that never fire on a batch of states,
• the effective rank of the last hidden layer (srank, Kumar et al., 2021), meaning how many directions the features really use out of 128,
• the weight norm.

At 226M steps, 21% of the plain network’s units were dead, srank was down from about 117 to 58, and the weight norm had grown from 24 to 194.

The proposed fix is continual backprop: keep replacing a trickle of the least useful hidden units with fresh ones. A unit’s usefulness is a running average of its activation times its outgoing weights. New units start with zero outgoing weights, so a reset changes nothing until the unit has learned something. I ran it on the same seed and config. It held dead units at 0.4% and srank at 116 for the full 226M steps, and a second seed showed the same. As a fix for the network’s health, it worked completely.

Whether that made the agent play better is less clear. On tier 7, the hardest tier any run held:

• Seed 0: the plain run had two long dips. It fell from 59% to 17% between 40M and 100M steps, and from 59% to 36% after 200M. The continual-backprop run stayed between 50% and 65% the whole time after its first 10M steps. From 100M to 200M, the two were level.
• Seed 1: the plain run never dipped. It averaged 63% on tier 7 over 100M steps, slightly above continual backprop’s 58%, while its dead units still climbed to 14%.
• Easier tiers: on tiers 0 to 6, all four runs averaged 74 to 76%.

After seed 0 alone, I was ready to write “continual backprop saves the agent”. Seed 1 is why that sentence isn’t in this article. What I think the evidence supports is narrower: plasticity loss is real and continual backprop removes it, but a network with a fifth of its units dead can still play well for a long time. The decay only sometimes turns into worse play. In my reading, continual backprop protects against the dips, but two seeds can’t tell me how often the dips happen.

Neither version ever held tier 8. So whatever stops the agent at tier 7 is not plasticity loss alone. My guess is network capacity or the observation, but I haven’t tested either.

What I’d take from this

• Measure the plain version first. The most effective mechanism in the system was a reservoir sample and a 15% chance of replaying an old tier. The clever ones needed an ablation to show they were costing me.
• Local metrics can all look good while the global one doesn’t move. The death bank fixed 23,117 failures. It still hurt.
• Run the second seed. One seed gave me a clean story about continual backprop, and the second one took most of it away.
• For timing-critical control, act with averaged weights. A network that changes a hundred times during an episode plays worse than any frozen copy of itself.
• What the agent can sense comes before everything else. No learning trick got it past gaps it couldn’t locate.

The open problem I’m left with is tier 7. Seven tiers took 2.4 hours, and the eighth hasn’t fallen in 74 more. I’d like to know whether it’s the 128-unit network, the 155-number observation or something about the training setup, and that’s the next experiment.

References

• Mnih et al., “Human-level control through deep reinforcement learning”, Nature 2015.
• van Hasselt, Guez and Silver, “Deep Reinforcement Learning with Double Q-learning”, AAAI 2016.
• Schaul, Quan, Antonoglou and Silver, “Prioritized Experience Replay”, ICLR 2016.
• Ecoffet, Huizinga, Lehman, Stanley and Clune, “First return, then explore”, Nature 2021.
• Florensa et al., “Reverse Curriculum Generation for Reinforcement Learning”, CoRL 2017.
• Jiang, Grefenstette and Rocktäschel, “Prioritized Level Replay”, ICML 2021.
• Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks”, PNAS 2017.
• Dohare, Hernandez-Garcia, Lan, Rahman, Mahmood and Sutton, “Loss of plasticity in deep continual learning”, Nature 2024.
• Lyle et al., “Understanding plasticity in neural networks”, ICML 2023.
• Kumar, Agarwal, Ghosh and Levine, “Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning”, ICLR 2021.
• Schaul et al., “The Phenomenon of Policy Churn”, NeurIPS 2022.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI