·Python · PyTorch · ViZDoom · CUDA

Deep RL plays Doom

A from-scratch PPO agent that learns to play Doom from raw pixels — and Doom you can play yourself.

The questions

Can a neural network, seeing only the screen, learn to play Doom well — from nothing but a reward signal?
And can it learn a whole level? Not a scenario arena — E1M1, the opening map of Knee-Deep in the Dead, spawn to exit switch.

▶

The agents playing

Three agents, one from-scratch PPO, the same pixels-in reward-out setup — but very different jobs. One holds its ground; one has to move; the last has to find its way out of a whole level.

The turret — defend the center: rooted in place, it learns to spin and shoot as monsters close in from every side (return ≈ 11).

The explorer — deadly corridor: it has to move, fighting its way down a hallway of enemies to reach the armor at the far end.

The level runner — E1M1, Knee-Deep in the Dead: a whole level, spawn to exit switch, in real time. Sixty million steps of straight PPO reached the exit zero times, so this agent was taught the level backwards — started beside the exit switch and moved one waypoint further back each time it succeeded, along a route recorded from a single human playthrough.It now finishes the level in about 19 seconds — the human it learned from took 168.It exits 19 times out of 20, with 88–101 health left and, almost always, zero kills. Nobody told it to do that: a kill is worth +1 and the exit +100, so fighting risks the only prize that matters. It rediscovered the speedrunner’s answer — run everything, shoot nothing, hit the switch.

◇

Inside the build

Each stage of the calculation. Click one to expand its full write-up and validation figure.

01the environment — pixels in, actions out

ViZDoom exposes the Doom engine as an RL environment. The agent sees only the framebuffer — grayscaled, downsized to 84×84, with the last 4 frames stacked so a single observation carries motion. Frame-skip repeats each chosen action for 4 tics, which is both faster and easier to learn from. On defend_the_center the reward is +1 per kill and −1 on death.

↳ feeds observations → the policy

02parallel environments — feeding the GPU

On-policy RL is data-hungry, so 16 Doom instances run at once. The naive version stepped them one-by-one at ~11 steps/s; because ViZDoom’s engine step is a C++ call that releases the Python GIL, stepping them in a thread pool runs the games concurrently and hits ~115 steps/s — a 10× speedup that turns a 2-day run into a few hours. A second, stranger lever showed up later: on Windows the scheduler’s default 15.6 ms tick quantises the engine’s per-frame wait, so an action that costs 3 ms of real work still burns a full 15.6 ms slice. The tell was that the rate was identical at every resolution — flatness means a clock, not rendering. One call to timeBeginPeriod(1) took twelve environments from 265 to 2038 steps/s, an 8× speedup that came from the operating system, not the code.

↳ feeds rollouts → PPO

03the policy network — a convolutional actor-critic

The classic “Nature CNN” — three convolutional layers into a 512-unit dense layer — reads the stacked frames and splits into two heads: a softmax over actions (the policy) and a single value estimate (the critic that judges how good the current state is). About 1.7 M parameters.

↳ feeds action + value → PPO

04PPO — the learning algorithm

Proximal Policy Optimization, written from scratch: collect a short rollout, estimate how much better each action was than expected with Generalized Advantage Estimation, then take a few gradient steps on the clipped surrogate objective. The clip keeps each update from moving the policy too far — the trick that makes PPO stable where naive policy gradients thrash. An entropy bonus keeps it exploring.

↳ feeds gradient updates → the policy

05training & recording — watching it learn

The same PPO trains two agents on a single GTX 1080. The turret (defend_the_center) goes from dying instantly to holding the center, return climbing ≈ 0 → 11 over 1.5 M steps. The explorer (deadly_corridor) has a richer action set — advance, strafe, turn, shoot — and must fight down a hallway to reach armor; over 3 M steps it cracks the corridor around 500k steps. Because ViZDoom reports the agent’s exact position each frame, its whole trajectory is logged and drawn from above — a top-down “God view” of it reaching deeper into the level as it learns.

06a whole level — why E1M1 is a different problem

The scenarios are arenas: a few seconds long, with reward arriving constantly. E1M1 is a real level — rooms, a courtyard, a zigzag, doors, an exit switch two minutes of walking away — and it pays almost nothing along the way. One bonus, at the end, hundreds of decisions after the choices that earned it. Straight PPO ran 60 M steps on it and reached the exit exactly zero times: the return curve climbed the whole way, but on exploration novelty, not progress. A rising score with nothing behind it is the failure mode this project spent the longest learning to detect, and the evaluation script now refuses to report a score without reporting how each episode actually ended.

↳ feeds the credit-assignment problem → the curriculum

07the reverse curriculum — teaching the last mile first

So the level was taught backwards. A human run of E1M1 is recorded once and thinned to 66 waypoints, one every 300 map units. The agent starts at the second-to-last waypoint — a few steps from the exit switch, where the bonus is easy to stumble into — and once it exits reliably, the start moves one waypoint further back. Every stage begins in territory the policy has already solved. Progress along the route is shaped with a potential-based term, the undiscounted difference between waypoints, which telescopes: backtracking hands the reward straight back, so a whole episode is worth at most the value of the route however it is walked, and there is nothing to farm. It worked immediately — 68 exits in the first 15 000 steps, against none in 60 M without it — and the curriculum walked all 66 stages back to the real map start in 14 M.

↳ feeds a policy that can finish the level → the forgetting problem

08keeping the policy — the run that solved it, and nearly lost it

The curriculum worked and then the agent kept forgetting it — exiting twenty episodes out of twenty, collapsing to zero, relearning, collapsing again, for 45 M steps. Two causes, both visible in the logs rather than guessed at. First, the entropy bonus: measured across the checkpoints, entropy averaged 0.16 where the agent was exiting ≥ 90 % of episodes and 1.00 where it exited ≤ 10 %, and it rose BEFORE the exits stopped — once the policy sharpens, advantages shrink, and a fixed exploration bonus outmuscles the policy gradient and drags it back toward random. Annealing it narrowed that swing to 0.08 against 0.31. Second, the curriculum’s own anti-forgetting trick switched itself off exactly when it was needed: the sampling window runs from the current stage forwards, and the stage counts DOWN, so on reaching the real map start it was only ever sampling the five hardest starts and had stopped revisiting the whole back half of the level. It showed: the agent could no longer finish from ten waypoints out, a stage it had cleared in its first few minutes. The decisive fix was neither of those, though — it was to stop keeping only the newest checkpoint and snapshot the best one, judged on real-map-start episodes alone. The very next run earned that snapshot seventeen minutes in, at 25 of 25 exits, and then spent four more hours oscillating and ended, once again, unable to finish the level at all.

◇

Figures

Click any figure to enlarge. Use ← → to move through the set.

◇

What it shows

Yes — with nothing but pixels and a reward signal, PPO learns a genuinely competent Doom policy. The learning curve is the clean, monotonic climb you hope for, plateauing near return 11 after about a million steps.
The engineering matters as much as the algorithm. The single biggest lever here was not a hyperparameter but realizing ViZDoom releases the GIL, so the environments could be stepped in parallel threads for a 10× throughput win — the difference between a feasible run and a two-day one.
The same agent, two very different skills. A turret that learns to hold ground and an explorer that learns to navigate and fight down a corridor come out of the identical PPO — the difference is only the action set and reward. And because ViZDoom exposes the agent’s position, the explorer’s learning can be watched in space: early paths die at the spawn, later ones push all the way to the exit.
Written from scratch — the CNN actor-critic, GAE, and the clipped PPO objective are all in ~200 lines, no RL library — so the whole pipeline is legible end to end: pixels in, a Doom-playing policy out.
A whole level needs curriculum, not more compute. Sixty million steps of straight PPO on E1M1 found the exit zero times; the same algorithm, handed a human route to walk backwards, was exiting within fifteen thousand. The lesson generalises past Doom: when reward is one event at the end of a long horizon, the fix is usually to move the starting line, not to train longer.
The level is solved: 19 exits in 20 attempts from the real map start, in about 19 seconds a run. The human demonstration it learned the route from took 168 seconds, so the student beat the teacher by roughly ninefold — and then stopped needing the lesson. In two of ten recorded runs the agent leaves the human’s route entirely and still reaches the exit; it found a shortcut the demonstration never took, which the route-progress metric scores as failure right up until the exit bonus lands.
It solved the level by refusing to play it. The agent finishes with 88–101 health out of 100 and, almost always, zero kills — it does not fight anything, it runs. That was never specified; it falls out of the arithmetic. A kill pays +1 and the exit pays +100, and dying forfeits the 100 outright, so combat is a bad trade twice over while speed is nearly free. What emerged is exactly what human Doom speedrunners do on this map. And it is the same network that learned, on defend_the_center, to stand still and shoot everything that moved — because there the reward paid per kill and there was no door to run to. Identical algorithm, opposite temperament, and the only thing that differed was what the reward paid for.
The most useful engineering lesson was the least clever one. This agent oscillates: it learns to finish the level, forgets, relearns, forgets, indefinitely. Annealing the entropy bonus dampened that considerably and did not stop it, and the run that produced the result above still ended, four hours later, unable to finish the level at all. What saved it was keeping a separate snapshot of the best policy — scored only on episodes from the real map start — instead of the most recent one. The final checkpoint of that run exits 0 times in 12. The snapshot taken seventeen minutes in exits 19 times in 20. On a task whose learning curve is not monotonic, deciding what to keep matters as much as what to train.

▶

Now you play

The same game, for you — the original Doom (shareware episode 1, “Knee-Deep in the Dead”), running in your browser through a self-hosted DOS emulator. Nothing is installed or streamed from elsewhere; it opens in a new tab so it has the whole screen.

▶Play Doom ↗ opens in a new tabWASD move · mouse aim · click to shoot · Space opens doors · click the game to capture the mouse