I Replaced the LLM Playing Mario With a Network That Learned From Scratch
Jev could clear World 1-1, but every move was a 600 ms API call. I copied it into a local model, then trained a PPO agent from raw pixels. It cleared the level 10 times out of 10, then got worse.

In my last Mario post, Jev, a language model behind an API, finally cleared World 1-1. Each clear took 208 decisions.
Every one of those decisions was a network call that took about 600 ms. The game paused while Jev thought. A level took minutes of wall time and 208 paid requests, and the logs showed it making the same choices run after run.
So I asked a different question: does Mario need a language model at all?
I tried two answers. First, copy Jev into a small local model. Then train a neural network from nothing but the screen and a score. I worked through both with Claude Code.
The cheap answer: copy Jev
Every Jev decision was already logged: the game state it saw and the action it chose. That is a supervised learning dataset. I had 2,127 rows.
The labels were lopsided. Jev picked right_jump 80.5% of the time, right_run 17.9% and right_run_jump 1.6%. It never used the other four actions. A model that always says right_jump would already be "80.5% accurate", so that was the baseline to beat.
I flattened each state into features, trained a random forest, and tested it on whole runs it had never seen:
| Model | Accuracy on held-out runs |
|---|---|
Always guess right_jump |
80.5% |
| Random forest | 98.8% |
That 98.8% is flattering. Several of Jev's clears were identical, so held-out runs still contained copies of training rows. The honest test was letting the student play.
It cleared World 1-1 in 208 decisions with zero Jev calls.
This approach, called behavior cloning, has a hard ceiling: the student can only be as good as its teacher, and only in situations the teacher showed it. To go past that, I needed an agent that learns from the game itself.
Training a PPO agent from pixels
The second agent knows nothing about Mario. No game-state parser, no hand-written hazard detection, no teacher. It sees the screen and a reward, and has to work out the rest.
I used PPO (Proximal Policy Optimization) from Stable-Baselines3 on gym-super-mario-bros.
What the network sees
The NES screen is 240×256 pixels in colour. That is far more than the agent needs:
NES screen 240×256 colour → shrink to 84×84 → grayscale → stack the last 4 frames
= 84×84×4 input
Four frames, because a single frame can't show motion. From one picture you can't tell whether Mario is rising or falling, or which way a Goomba is walking. Four in a row show it.
The agent also chooses only every 4th frame and repeats its action in between. Consecutive frames are nearly identical, so this makes training roughly four times cheaper.
The network
It is the standard convolutional network from DeepMind's Atari work, about 1.7 million weights, with two outputs:
84×84×4 → conv → conv → conv → 512 features
├── actor: 7 probabilities, one per button combo
└── critic: 1 number, "how much reward do I expect from here?"
The actor decides what to press. The critic exists only for training: it is how the agent judges whether a move turned out better or worse than expected.
What it is rewarded for
The environment's built-in reward is simple:
reward = pixels moved right − a small penalty as the clock ticks − 15 if Mario dies
Nothing for coins. Nothing for enemies. Remember that; it matters later.
How the PPO training loop works
All of PPO training is one loop. My run repeated it 733 times:
1. PLAY 8 games × 512 moves = 4,096 moves with the current network. Record everything.
2. SCORE For each move: was it better or worse than the critic expected?
3. LEARN Nudge the network toward the better moves, by a small, capped amount.
4. REPEAT Throw the data away and play again with the updated network.
733 rounds × 4,096 moves comes to 3 million moves.
1. Play
Eight copies of the game run in parallel. At each step the network outputs probabilities for the 7 actions, and the action is sampled, not picked. If right_jump has probability 0.71, it gets chosen 71% of the time. That randomness is how the agent discovers that something else might work better.
Each step is stored: the screen, the action, its probability, the critic's guess, the reward, and whether the game ended.
2. Score: the advantage
Reward alone is a poor teacher. A jump at move 100 might only pay off at move 110, when Mario lands safely past a pit. So instead of asking "was this reward high?", PPO asks: was this better than expected?
For every move:
surprise_t = reward_t + 0.9 × value(next state) − value(this state)
advantage_t = surprise_t + 0.855 × surprise_t+1 + 0.855² × surprise_t+2 + …
This is called Generalised Advantage Estimation. In plain words: add up the surprises that followed a move, counting the near ones fully and fading out the far ones.
The 0.9 in that formula is gamma, how far ahead the agent cares. A reward 10 moves later counts for about a third. Mario is a reflex game, so a short horizon is enough, and it makes learning faster.
If Mario jumps over a Goomba and keeps running, the advantage is positive: do more of that here. If he jumps into it, the death penalty and the lost progress make it strongly negative.
3. Learn: the clipped update
The 4,096 moves are shuffled into 16 batches of 256, and the network trains on them for 10 passes, 160 gradient steps per round. Each step reduces one combined loss:
loss = policy loss make better-than-expected moves more likely
+ 0.5 × value loss make the critic's predictions match what happened
− 0.01 × entropy a bonus for staying a little random, to keep exploring
The policy loss is where PPO earns its name. For each move it compares the new network with the one that actually played:
ratio = P_new(action) / P_old(action)
A good move pushes the ratio up, a bad one pushes it down. But the ratio is clipped to 0.8–1.2. Past that range, pushing further earns nothing. No single batch, however lucky, can change the policy by much in one round.
That clip is the whole safety idea behind PPO. Older methods without it could wreck a good policy with one oversized update.
After 160 steps, the data is thrown away. It was collected by the old policy, so reusing it further would mislead the new one. Then the loop starts again.
The settings
PPO("CnnPolicy", env,
n_steps=512, # moves per game per round (8 games → 4,096)
batch_size=256, # moves per gradient step
n_epochs=10, # passes over each round's data
learning_rate=2.5e-4, # size of each gradient step
gamma=0.9, # how far ahead rewards count
gae_lambda=0.95, # trust real rewards vs the critic's guesses
clip_range=0.2, # max policy change per round
ent_coef=0.01) # exploration bonus
Results: testing every checkpoint
I saved a checkpoint every 400,000 moves, then tested each one two ways:
- Fixed: always take the most likely action. Same start, same choices, so the same run every time.
- Sampled: pick actions by their probabilities, 10 runs per checkpoint. This tests whether it can recover when things go slightly differently.
The flag is at x ≈ 3,161.
| Checkpoint | Fixed | Clears | Avg x |
|---|---|---|---|
| 400k | 1,795 | 2/10 | 1,992 |
| 800k | 2,979 | 5/10 | 2,421 |
| 1.2M | Flag | 8/10 | 2,664 |
| 1.6M | Flag | 7/10 | 2,915 |
| 2.0M | Flag | 7/10 | 2,911 |
| 2.4M | Flag | 10/10 | 3,161 |
| 2.8M | Flag | 3/10 | 2,057 |
| Final | 2,465 | 4/10 | 2,383 |
"Fixed" is the fixed run: Flag means it reached the flag, a number is how far it got. "Clears" and "Avg x" are from the 10 sampled runs. "Final" is the model at 3.0M moves.
The early checkpoints tell the expected story. At 400k moves, it had learned that right is good and could get past the first pipes and enemies. At 800k it got within 200 pixels of the flag. At 1.2M it cleared the level for the first time.
At 2.4M it peaked: ten clears out of ten, even with sampled actions.
Then it got worse. By the end of training it no longer cleared the level on its fixed run.
Why the final model wasn't the best one
Once the agent clears the level most of the time, nearly every move looks about as good as expected. The advantages shrink toward zero and the updates become mostly noise. A few unlucky rounds can push the policy toward a slightly worse route.
The clip limits how far each update moves. It does not guarantee the move is in the right direction.
The fix is boring and important: evaluate every checkpoint and keep the best one. If I had only tested the final model, I would have concluded that 3 million moves of training produced an agent that fails World 1-1. It didn't. It produced a good agent at 2.4M, then trained past it.
Lowering the learning rate over time, or evaluating during training and saving the best checkpoint automatically, would both help.
Two things it doesn't do
It plays the same way every time. The NES emulator is fully deterministic. With the same start and the most likely action at every step, the agent replays the identical run. Sampling actions brings some variety, but it is still one learned route with small wobbles.
It ignores coins. Watching it, the first thing I noticed was that it ran past every coin. Of course it did. Coins are worth nothing in its reward, and a detour costs time. The agent learned exactly what I paid it for.
That's the most useful thing this experiment taught me. Jev collected coins because its prompt and reward said to. The PPO agent never will unless the reward says so. Changing what it learns means changing step 2 of the loop, and nothing else.
Jev, the copy, and the learner
| Jev | Copy | PPO | |
|---|---|---|---|
| Learns | Pre-training | Jev's logs | Reward |
| Moves | API call | Local | Local |
| Clears | Yes | Yes | 10/10 |
| Coins | 1 per clear | Not tested | None |
| Teacher | None | Jev | None |
Each one is good at something different. Jev brings general knowledge and follows instructions. The copied student is nearly free to run and plays like Jev on the situations Jev showed it. The PPO agent needs no teacher at all, but optimises exactly the number you give it, nothing more.
Next, I want to add a coin reward and fine-tune from the 2.4M checkpoint. Then I'll find out how much it likes coins compared with not dying.