> ## Content Index
> Fetch the complete content index at: https://iadar.sh/llms.txt
> Use this file to discover other available public pages before exploring further.

# Replacing Jev With an Open Model in Super Mario Bros. Took Four Tries
- URL: https://iadar.sh/laya-mario-open-model/
- Published: 2026-10-05T11:47:35.000Z
- Updated: 2026-10-05T11:47:35.000Z
- Description: I swapped the Jev API for Laya, an open 421M decision model, in my Mario controller. Out of the box it died at the first Goomba. After a shorter prompt, two fine-tunes and a fallback, it cleared World 1-1 with 7 Jev calls instead of 208.
- Author: Adarsh Pandey
- Tags: machine-learning, open-source

My Mario controller asks [Jev](https://iadar.sh/jev-mario-controller-learning/) what to do roughly every 8 frames. Each answer is an API call that takes about 600 ms, and a clear of World 1-1 takes 208 of them.

In the [last post](https://iadar.sh/mario-ppo-from-pixels/) I got rid of those calls twice: once by copying Jev into a small random forest, and once by training a PPO agent from pixels. Both work, but both are narrow. The forest only knows Mario's state as numbers, and the PPO agent only knows pixels.

Then I found [Laya](https://huggingface.co/convaiinnovations/laya?ref=iadar.sh). It is an open-weights model (Apache 2.0) built for the same job as Jev: give it a state and typed questions, and it returns an answer with probabilities in one forward pass. Its model card compares itself directly with Jev, and it ships a server that speaks Jev's API.

That looked like a drop-in replacement. It wasn't, at first. Here is every experiment, including the ones that failed. I worked through them with Claude Code.

## Laya in one paragraph

Laya is a 421M-parameter encoder (ModernBERT-large plus a small decision head). It doesn't generate text. Each answer option gets a marker token, and the model scores all options at once. The model card claims about 33 ms for one question on a GPU, and it is candid about one limit: the base checkpoints are "near chance zero-shot" on new decision types and need fine-tuning to be useful.

I didn't take that seriously enough at first.

## Try 1: point the controller at Laya

Because Laya's server accepts Jev's request format, the controller needed no code changes. I pointed its base URL at the local server and ran it with the exact same prompt, state and seven controller actions.

I scored it two ways: by replaying 200 of Jev's logged decisions to see how often Laya picked the same action, and by letting it play.

| Checkpoint   | Agrees with Jev | Game            |
| ------------ | --------------- | --------------- |
| English      | 13%             | Died at x 1,935 |
| Multilingual | 25%             | Ended at x 171  |

The flag is at x ≈ 3,161\. Each decision took about 210 ms, not 33.

The cause showed up in the server's usage report:

```text
state_tokens: 680   state_tokens_dropped: 364   truncated: true
state_tokens: 897   state_tokens_dropped: 581   truncated: true

```

The English checkpoint reads 512 tokens. Jev's prompt, written for Jev, has three questions with long instructions plus a large JSON state. Laya was dropping 40–65% of the game state on every call, including the enemy and terrain fields it needed most.

## Try 2: a prompt that fits

I wrote a separate Laya policy with a compact state: only the fields a decision depends on (Mario's position, speed and jump phase, the nearest enemy and time to contact, the next obstacle or gap), and a single one-line question.

To test it fairly, I held out Jev's two most recent runs (624 decisions) and never trained on them.

| Prompt             | Agrees with Jev | Per decision |
| ------------------ | --------------- | ------------ |
| Jev's prompt       | 13%             | \~210 ms     |
| Compact            | 70%             | 72 ms        |
| Always right\_jump | 85%             | –            |

Much better, and three times faster. But still worse than a model that presses `right_jump` every time, because Jev presses it 85% of the time. In the game it reached x 1,216 before dying.

That's the zero-shot ceiling the model card warned about. Next step: teach it.

## Try 3: fine-tune on Jev's decisions

Every Jev decision was logged with the state it saw and the probability it gave each action. That's a training set: 1,503 decisions from the older runs. I used Laya's own fine-tuning script unchanged and fed it my items, with Jev's full probabilities as soft targets rather than just its top choice.

Four epochs took about 40 minutes. The result looked great:

| Model                | Agrees with Jev |
| -------------------- | --------------- |
| Base, compact prompt | 70.2%           |
| Fine-tuned           | 93.8%           |

Then it played, and died at x 309\. The first Goomba.

Here's what it saw on its last few decisions:

```text
x 261  goomba 69 px ahead, contact in 23 frames    right_run 0.71   right_jump 0.15
x 285  goomba 40 px ahead, MUST JUMP NOW           right_run 0.69   right_jump 0.15
x 309  goomba 11 px ahead, contact in 4 frames     right_run 0.70   right_jump 0.15

```

The probabilities barely moved. At x 285 the state said the jump had to start on this decision, and it still chose to run. In Jev's logged runs, it pressed `right_run_jump` at exactly that point.

### Why 93.8% was a lie

Most of Jev's decisions are easy: keep holding jump while in the air, keep running on open ground. The decisions that matter, taking off from the ground at the right moment, are rare. Only 117 of the 1,503 training rows (7.8%) were takeoffs.

So I added a second metric: of the 39 held-out moments where Jev jumped from the ground, how often did Laya also jump?

| Model         | Agrees | Takeoffs |
| ------------- | ------ | -------- |
| Base, compact | 70.2%  | 33/39    |
| Fine-tuned    | 93.8%  | 15/39    |

Fine-tuning made it better at the easy decisions and worse at the one that keeps Mario alive. Overall accuracy hid it completely.

## Try 4: make the danger readable, and make it count

Two changes.

**A plain-language summary.** Laya is a language model. A `true` buried among two dozen JSON fields is hard for it to notice; a sentence isn't. Every state now starts with one line built from the same fields:

```text
on the ground; goomba 40 px ahead, contact in 13 frames; MUST JUMP NOW
in the air, rising; gap 7 tiles wide, 2 tiles ahead; crossing a gap, keep moving right

```

**Weight the takeoffs.** Each ground takeoff appears 8 times in the training data, so the model can't score well by ignoring them. That made 2,322 training items; I trained for two epochs.

| Model         | Agrees | Takeoffs |
| ------------- | ------ | -------- |
| Fine-tuned v1 | 93.8%  | 15/39    |
| Fine-tuned v2 | 99.5%  | 39/39    |

It cleared the Goombas and the pipes on Jev's route. Then it died at x 1,134, the first pit.

### Dying at the pit

This time the decision wasn't wrong. It was the right decision from the wrong place.

In Jev's logged runs, Mario lands from the previous jump at x 1,048 and take off again from there. Laya's earlier jump came down a little sooner, at about x 1,020\. It did what Jev does on landing, jump again, and came up short.

This is the standard weakness of copying a teacher. The student only learns the states the teacher visited. Drift a few pixels off that path and it's somewhere the teacher never showed it, and small errors compound.

## The fix that worked: ask Jev when unsure

The forest from the last post has a safety net: when it isn't confident, it asks Jev. I gave Laya the same one. If Laya's confidence is below 0.5 and Mario is on the ground (a takeoff decision), Jev decides that move.

The first attempt still died at the pit, with 8 Jev calls. Allowing Jev to decide in the air as well raised it to 32 calls, and it still died at the same pit.

That was suspicious, so I diffed this run against one of Jev's clears. They were identical until decision 19, at x 373\. Same position, same state. In the clearing run Jev chose `right_run`. Asked from the Laya run, Jev chose `jump` with 0.13 confidence, then `noop`, and the run drifted from there.

The difference wasn't in the logs. Jev's clears came after I added **experience memory**: a summary of earlier attempts in the same situation, sent along with each request. The logs record the game state but not that memory. My fallback was calling Jev without it.

With the fallback Jev given a copy of its memory, and every move's outcome (Laya's included) fed back into it:

|                | Jev alone | Laya + Jev |
| -------------- | --------- | ---------- |
| Decisions      | 208       | 208        |
| Jev API calls  | 208       | 7          |
| Laya decisions | 0         | 201        |
| Result         | Clear     | Clear      |

It reached the flag twice in a row, the same way both times. Jev was called at x 331, 373, 1,022, 1,432, 1,565, 2,691 and 3,143\. The call at x 1,022 is the takeoff before the pit that killed every earlier version. Laya's own decisions took about 66 ms each.

## What I took from this

**"Drop-in compatible" covers the API, not the prompt.** Laya accepts Jev's requests, but a prompt written for Jev silently overflows a 512-token encoder. Check truncation before judging any model.

**Read the model card's limits.** "Near chance zero-shot" was right there. The model was a starting point to fine-tune, and I treated it as a finished product.

**Overall accuracy can hide the decision that matters.** 93.8% looked like success and died at the first enemy. Measure the rare, high-stakes cases separately.

**Write the important signal so the model can read it.** A one-line summary did more than a careful JSON schema.

**Log everything the teacher saw.** The hardest bug was a memory the logs didn't record. Jev with memory and Jev without it are different teachers.

## Where the three local approaches stand

|             | Forest     | PPO    | Laya       |
| ----------- | ---------- | ------ | ---------- |
| Learns from | Jev's logs | Reward | Jev's logs |
| Input       | Numbers    | Pixels | Text       |
| Per move    | <10 ms     | –      | \~66 ms    |
| Jev calls   | 0          | 0      | 7          |

"Forest" is the random-forest copy of Jev. I didn't time the PPO agent's moves, and Laya's 7 Jev calls are per clear.

On this one level, the random forest is still the cheapest. Laya's advantage is that it reads text, so the same model could take new instructions without new features.

The next step is DAgger: let Laya play, have Jev label the states Laya actually reaches, retrain, and repeat until it no longer needs the 7 calls.