The idea
One process, three games
DOS gives a program the whole machine and nothing else: no processes, no threads, no memory protection. Doom 2 was written on the assumption that it is the only thing running, and its state is a few hundred kilobytes of globals at fixed addresses, so running three of it means doing by hand what an operating system would do: give each copy memory of its own, save and restore the parts they have to share, and move the CPU between them.
The switch happens every four frames, at the end of a frame, so a copy is never stopped in the middle of rendering or thinking. A copy's share of the common state is 577,494 bytes, but in the recorded run only 1,336 of them differed at a switch, so the host compares everything and copies just those.
Each copy draws into its own framebuffer, which means one copy can use another's framebuffer as a wall texture. Every level has such a wall showing the next copy, and the last one shows the first, so the picture loops.
The model
How it works
There is one copy of Doom's code, shared by every instance, which the project calls a depth. Each depth has its own heap, stack, screen buffer and copy of the ACE code at addresses of its own, but Doom's data sits at the fixed addresses the code was compiled against, so only one depth's data can be live at a time. A switch saves the outgoing depth's data, loads the incoming one's, points three pointers at the new depth's heap and screen, and hands over the CPU.
Up close
Anatomy
A depth
What makes a depth a separate game is memory of its own plus a saved copy of the shared state it has changed. Pick a depth to run and everything that has to follow it re-points.
- Data. Doom's globals exist once. Most of them are tables that never change after startup, so they are the same in every depth, and the few bytes that do differ (coloured) belong to whichever depth is running.
- Pointers. Three of those globals must not come back from a snapshot:
mainzone, ACE'sscreen_bufferand Doom'sscreens[0]. The host pins them to the running depth's heap and screen after every load. It takes three and not two because Doom's renderer writes through its ownscreens[0], not through ACE's copy. - Patches. ACE works by patching Doom, at 406 places, nearly all of them in the code segment, which no data snapshot covers. Each depth's ACE points those patches at its own copy of the ACE code, so the patch bytes are saved and restored per depth exactly like data.
- Host state. Which depth runs, and where every suspended depth's stack pointer is, live in one object that no snapshot includes. The first version kept them in the same
.bssit was snapshotting, and the first real switch restored a stack pointer saved before it was written and jumped with ESP = 0.
In the code: nested/dos_wire.c (mtx_dos_wire_register).
A switch
Every four frames the host saves the running depth, loads the next one, re-pins the pointers and swaps the stacks. Saving compares each registered block against the depth's snapshot and copies only the blocks that changed, and the stack swap is a handful of instructions: push the four callee-saved registers, store ESP, load the other depth's ESP, pop, return into wherever that depth last switched away.
In the code: nested/mtx_core.c (mtx_switch), nested/dos_state.c (mtx_dos_save_state, mtx_dos_load_state), nested/mtx_sched.c (mtx_frame_tick).
The pictures
Each depth renders into its own screen buffer, and the wall texture in its level samples the next depth's buffer, whose address the host hands it every frame (screen_src). The deepest depth's wall points back at depth 0. A suspended depth doesn't render, so what its neighbour shows is its last frame.
Only one buffer reaches the monitor: every depth's blit copies the viewed depth's, and y moves the viewer one depth along. The viewed depth also gets every other time slice, which keeps the picture you look at at 30 frames a second while the others run at 15.
In the code: engine/mtx_boot.c (mtx_ring_next, mtx_world_screen, mtx_engine_video, mtx_sched_next).
The code
Where it is
Everything is in one repository, a fork of kgsws' doom_ace, with the engine at the root and the switching in nested/. The README there goes through how each piece was built, and what broke along the way.
| Part | Code | Background |
|---|---|---|
| Handing over the CPU | nested/mtx_switch.S (mtx_switch_ctx), nested/mtx_core.c (mtx_switch_to) | README, “The mechanism” |
| When it switches | nested/mtx_sched.c (mtx_frame_tick), engine/mtx_boot.c (MTX_DEMO_INTERVAL, mtx_sched_next) | README, “The yield point” |
| Swapping data, patches, pointers | nested/dos_state.c (mtx_dos_save_state, mtx_dos_load_state), nested/mtx_hooks.c | README, “What is per-depth” and “405 patches” |
| The switch's own state | nested/mtx_host.h (mtx_hs) | README, “Live switching, and the illegal-read fix” |
| A depth's own memory | nested/mtx_zone.c, nested/mtx_raw.c, stack slots in nested/mtx_core.c (mtx_seed) | README, “A depth's own zone” |
| Starting a depth | nested/ldr0N.S, nested/ldr1n.c, engine/mtx_ldr.c | README, “The loaders” |
| The walls and the loop | engine/mtx_boot.c (mtx_ring_next, mtx_screen_paint), nested/mtx_params.h (screen_src) | comments in that code |
Measured
Your turn
The challenge
The sandbox is a small version of the same problem that you can solve yourself: a made-up 64 KiB computer, a C compiler for it, and a tiny Doom-like game whose level scripts patch its code. You write the scheduler that lets several copies of the game share the machine, and a checker runs every level alone first, then compares each copy's state against that solo run, frame by frame:
- Take turns.
- Keep their data apart.
- Keep their patches apart.
- Keep their pointers apart.
- Survive a hostile level.
- Close the ring.
Once a scheduler passes, it is scored on the cycles and bytes each switch costs, which is where the real optimisation work starts.
The page compiles your scheduler, checks it, and lets you step the machine one instruction at a time. Its compiler, machine and checker are the kit's own C compiled to WebAssembly, so the results are the same as make check in sandbox/.