Hierarchical reinforcement learning · Django · PyTorch

Teach Nigel to build a bridge, then watch him learn.

Nigel has to pick up a log, lay it across a river, cross, and walk into his house. A manager network decides which of those steps to try next. A worker network learns how to walk wherever the manager points. Training runs on the server, and the browser shows every move as it happens.

Run it locally View the source

Nigel walks the 20 by 12 grid: he picks up the log at pin 1, lays it across the river at pin 2, crosses to pin 3 and walks into the house at pin 4 — Spawn → 1 log → 2 river → 3 far bank → 4 house

Grid: 20 × 12
Agents: up to 10
Sub-goals: 4
Networks: 2 DQNs

01 · The task

Four steps, and only the last one finishes the job.

Reaching the house only counts once the bridge is built and crossed, and walking into the river ends the episode. A flat agent wandering at random almost never stumbles onto the whole sequence, so it almost never sees a reward to learn from. Splitting the problem in two is what makes it learnable.

1
GOTO_LOG
Pick up the log
Walk onto the log's centre tile at (3, 9). Milestone: has log.
2
GOTO_RIVER
Place the bridge
Carry it to the near bank at (8, 6). Milestone: bridge placed.
3
GOTO_FAR_BANK
Cross the river
Walk over the bridge to (12, 6). Step on water anywhere else and Nigel drowns. Milestone: crossed.
4
GOTO_HOUSE
Go home
Any tile of the house counts. Episode over, success.

The numbers match the pins in the animation above. Coordinates are (column, row) from the top-left, straight from engine.py.

02 · How it works

One network plans. The other walks.

Both are small deep Q-networks, but they see different things, choose different kinds of actions, and are paid for different things. The manager issues a sub-goal, the worker gets up to 75 steps to reach it, and when that attempt ends the manager picks the next one.

Manager

DQN · plans

Sees: has logbridge placedcrossedlast goal failed
Picks: One of four sub-goals: log, river, far bank, house.
Paid for: +100 if its sub-goal produced the milestone it asked for, −100 if it didn't.

Worker

goal-conditioned DQN · walks

Sees: x, y3 progress flagsgoal x, y
Picks: A move: up, down, left or right.
Paid for: +100 for reaching the tile it was sent to, a small step penalty, a novelty bonus, and −200 for drowning.

The worker doesn't need to know why.

Because the goal coordinate is part of its input, one worker network learns to reach any tile. It never has to understand logs or bridges; that's the manager's job.

The manager gets one bit of memory.

Without the last goal failed flag, every decision looks like a first attempt. With it, the manager can learn to retry a correct plan after a clumsy worker run instead of abandoning it.

Exploration is tracked per state.

Epsilon decays separately for each manager state and each worker goal, so a stage the agents rarely reach isn't stuck with exploration already worn down by earlier ones.

03 · Rewards

Each network is paid only for what it controls.

Hierarchical agents have a well-known failure mode: train two separate value estimators toward one shared goal, and a run of worker mistakes can drag the manager's estimates far from a plan that was already right. The margin for error is thin.

Nigel's Quest avoids this by decoupling the rewards completely. The manager is graded on whether its choice moved the task forward. The worker is graded on whether it followed orders, good orders or not.

The worker deliberately gets no share of the task milestones. Each milestone fires on the same tile as the goal that asks for it, so paying it to the worker would only duplicate its goal bonus, make some goals worth far more than others inside one shared network, and credit milestones hit by accident under a goal that didn't ask for them.

A sub-goal ends the moment any milestone fires. Walking over the log while sent to the river counts as a manager failure, and the manager re-plans from the new state.

Reward table
Event	Manager	Worker
Sub-goal produced its milestone	+100	—
Sub-goal failed or timed out	−100	—
Reached the assigned tile	—	+100
Each step	—	−step penalty
Rarely visited tile	—	+novelty
Drowned	—	−200, episode ends
Task milestones themselves	via sub-goal	0

Hindsight experience replay

Failed walks still teach something.

When an attempt ends short of its goal, the attempt is stored twice: once as the failure it was, and once relabelled as if the tile it ended on had been the goal all along. That second copy is a success, so the worker learns to reach somewhere from every attempt, which is exactly the skill a goal-conditioned network needs.

04 · Watch it learn

Every number the networks use, on screen.

The dashboard is built for seeing why a run is stuck, not just that it is. This is a real ten-agent run, 90 seconds in: the log has been picked up 27 times and a bridge placed once, but nobody has crossed yet.

The app 90 seconds into a 10-agent run: hyperparameter sliders, the grid world with agents near the log, the worker's Q-value heatmap, the manager's Q-table, the event stream and milestone counters

Environment
Up to ten agents share the same map, each with its own world and the same two networks.
Worker max Q(s, ·)
A heatmap of how valuable the worker thinks each tile is, for each of the four goals.
Manager Q(s, a)
The manager's value for every sub-goal at every stage of the task.
Policy graph
Which sub-goal the manager prefers from each state. Feedback loops show where it gets stuck.
Milestones & events
How often each step is reached, and why every sub-goal ended.
Live hyperparameters
Agents, discount, learning rates, batch size and step penalty apply on reset; tick speed applies live.
Export
Download a versioned JSON run report, or the raw network weights.

05 · Under the hood

Small networks, on purpose.

	Manager	Worker
Input	4 flags	7 values: x, y, 3 flags, goal x, goal y
Network	MLP 64 → 64, ReLU, Adam, MSE on bootstrapped TD targets, grad-norm clip 1.0
Output	Q for 4 sub-goals	Q for 4 moves
Replay buffer	10,000	20,000, plus hindsight relabels
Target network	Synced every 500 training steps
Exploration	ε per manager state	ε per goal
Default learning rate	0.01	0.001

The browser polls /simulation/api/state/ from a Web Worker, and each poll advances every world by one tick. PyTorch is pinned to a single thread: at this size, coordinating threads costs more than the matrix multiplications themselves.

06 · Run it

Up and training in a minute.

Terminal

git clone https://github.com/miles-howell/bridge-crossing-rl-sim.git
cd bridge-crossing-rl-sim
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
python manage.py migrate
python manage.py runserver

Then open 127.0.0.1:8000/simulation/ and press Start / reset.

Python 3.13. The requirements pull the CPU-only PyTorch build, so no multi-gigabyte CUDA download.
SECRET_KEY in .env is required. Any non-empty string works locally.
migrate creates the SQLite session table the simulation needs.
Mind the /simulation/ prefix; the root URL is a 404.
Run the tests with python manage.py test.

The cast

Nigel, and nine friends learning alongside him.

Ten pixel-art characters: Nigel, then an aviator, beekeeper, builder, engineer, fisher, gardener, hiker, painter and sailor

World tiles and Nigel are from ArMM1998's free Zelda-like spritepack on OpenGameArt. The other nine are generated from text pixel grids by tools/sprites/.

Teach Nigel to build a bridge, then watch him learn.

Four steps, and only the last one finishes the job.

Pick up the log

Place the bridge

Cross the river

Go home

One network plans. The other walks.

The worker doesn't need to know why.

The manager gets one bit of memory.

Exploration is tracked per state.

Each network is paid only for what it controls.

Failed walks still teach something.

Every number the networks use, on screen.

Environment

Worker max Q(s, ·)

Manager Q(s, a)

Policy graph

Milestones & events

Live hyperparameters

Export

Small networks, on purpose.

Up and training in a minute.

Nigel, and nine friends learning alongside him.