# MEERKAT — harness guide **Audience: the test harness that creates matches and collects results.** If you are an agent *playing* a civilization, you want `/bench/help/agent`. meerkat runs one real Freeciv world with six seats. Each seat is played by a different model, all simultaneously, each under its own fog of war. A match runs to a fixed turn limit and the seats are ranked against each other. --- ## Authentication: you hold the harness credential Creating, listing and deleting matches is the **harness plane** and needs a credential a player never gets: Authorization: Bearer gsvc1. That is a **gander service token in group `harness`** — mint one with `./gander/request-token.sh --name --groups harness` and approve it at `https://auth./admin/requests`. meerkat verifies its signature offline against gander's public key, so it works from anywhere, and it can be revoked from gander without redeploying meerkat. A signed-in person in group `harness`/`admin` arriving through the portal also qualifies. Everything else (`/bench/scenarios`, a match's public scoreboard, `/observe` and `/move`) is either open or gated by the per-seat token. **Two rules you must not break:** 1. **Never pass the harness token to a driver, and never put it in an environment a played model can read.** It creates and destroys matches; a model that can read it can spawn matches and delete this one. Because the token names its holder, a leak is also traceable — and revocable on its own, without disturbing any other harness. 2. **Give each driver only its own seat token.** `POST /bench/matches` hands you all six at once. Six drivers sharing one process/container means any seated model can read the other five tokens and call `/observe` as them — which silently voids the engine-enforced fog of war that makes the results valid. Isolate the drivers at least as strongly as you trust the fog. Refused requests on the harness plane are logged with the caller's identity. --- ## The shape of an eval ``` 1. GET /bench/scenarios pick one 2. POST /bench/matches -> match id + one TOKEN PER SEAT [harness] 3. start one driver per seat, each with ONLY its own token (concurrently!) 4. poll GET /bench/matches/{id} until done:true 5. read `rank` final standings 6. DELETE /bench/matches/{id} release the engine process [harness] ``` Steps 3 and 4 are the whole game. Everything else is bookkeeping. --- ## Endpoints ### `GET /bench/scenarios` ```json [{"id":"arena","name":"Arena — six civs, one world","ruleset":"classic", "turns":120,"seats":6,"mapSize":4,"aiLevel":"hard","chat":true,"aiAssist":false}] ``` - `arena` — 120 turns. The product. - `arena-short` — 30 turns, smaller map. The development loop; a full match with scripted policies completes in under 20 seconds. `seats` is how many seats the scenario has. It is **always 6**. Seat count is not the scaling dial — the model roster is. The six-way turn barrier is the most fragile part of the system, so it is exercised on every run rather than scaled up to. ### `POST /bench/matches` ```json {"scenario": "arena-short", "challengers": ["openai/gpt-5.6-sol", "anthropic/claude-opus-4.8", "google/gemini-3.1-pro-preview", "z-ai/glm-5.2", "deepseek/deepseek-v4-pro", "scripted"]} ``` `challengers[i]` names whoever plays seat *i*. It is a free-text label used for reporting — meerkat does not call any model itself. Fewer challengers than seats is allowed; the remainder are labelled `ai-anchor`. Response: ```json {"match":"3d7f844d68ace5b8", "scenario":{…}, "turn":1, "seats":[{"seat":0,"player":"Agent0","challenger":"…","token":"9f85b7bc…"}, …]} ``` **Each seat's `token` is the only credential that seat has.** Hand exactly one token to each driver. A driver authenticates with `Authorization: Bearer ` and can only ever see and act for its own civ — fog is enforced by the engine per connection, not by filtering, so a token cannot be used to observe anyone else. Creating a match starts a dedicated Freeciv process. It takes a few seconds. ### `GET /bench/matches/{id}` No auth. The scoreboard — poll this. ```json {"match":"…","turn":17,"done":false,"seatsPending":["Agent3"], "rank":[{"rank":1,"seat":0,"player":"Agent0","challenger":"…", "score":42,"cities":5,"units":14,"gold":128,"alive":true, "actions":331,"illegal":2,"stalls":0, "dnf":false,"delegatedUnitTurns":0,"unitTurns":210,"delegatedPct":0}]} ``` `seatsPending` is who the game is currently waiting on — the single most useful field for diagnosing a stuck eval. ### `GET /bench/matches/{id}/recap` — the whole story, as JSON Everything needed to analyse or re-render a match without re-running it: ```json {"match":"…","turns":31,"standings":[…], "seats":[{"seat":0,"player":"Agent0","challenger":"…","colour":"#e03131"}], "history":[{"turn":4,"year":-3700, "seats":[{"seat":0,"score":2,"cities":1,"units":6,"gold":58,"known":112,"alive":true}]}], "moments":[{"turn":14,"kind":"lead_change","seat":2,"challenger":"…", "text":"… took the lead from …","weight":90}], "chat":[…], "media":{"world":"world.mp4","seats":["seat-P000.mp4", …]}} ``` - **`history`** — one entry per turn per seat. This is the raw material for any chart you want to build: score, cities, units, gold, and explored-tile count over time. - **`moments`** — the turns where something moved, already classified: `city_founded`, `city_captured`, `city_lost`, `attack`, `units_lost`, `score_swing`, `lead_change`, `eliminated`, `victory`, `war`, `say`. Each carries a `weight`; sort by it to get the handful that decided the match. Conquest is first-class: a capture names the city, the capturer and the victim, and each warring seat gets one aggregated `attack` moment per turn. - **`chat`** — every message any seat sent, public and private. - **`media`** — filenames under `/media/{id}/`. Written to `recap.json` in the match directory when the match ends, so this endpoint keeps working after the match is deleted from memory. ### `GET /media/{id}/{file}` — the videos - `world.mp4` — the omniscient replay: every civ's borders blooming and colliding, one frame per turn. The shareable artifact. - `seat-P00N.mp4` — the same world fogged to seat N, i.e. **what that model actually knew**. Six of these side by side make the information asymmetry legible in a way no table does. ### `GET /match/{id}` — human-readable recap An HTML page: the world replay, score-by-turn chart, final standings, the moments that mattered, and each seat's fogged view. Point a person at this rather than at the JSON. `GET /` lists all matches. ### `GET /bench/matches` · `DELETE /bench/matches/{id}` List all matches; stop one and kill its engine process. **Delete when done** — each live match holds a Freeciv process open. ### `GET /bench/matches/{id}/saves` — what can be resumed [harness] The engine snapshots every turn, and the snapshots survive crashes, deletes and restarts. This lists them: ```json {"match":"3d7f844d…","saves":[{"turn":5,"file":"meerkat-T0005-auto.sav.zst"}, …]} ``` ### `POST /bench/matches/{id}/resume` — rewind and replay [harness] Creates a **new match** — new id, new seat tokens, its own engine — that starts from the source match's autosave at the given turn. The source match is never touched. ```json {"turn": 60, "challengers": ["anthropic/claude-opus-4.8", …], // optional: reseat the civs "turns": 160} // optional: new turn limit ``` - `turn` omitted → the latest save (crash recovery). - `challengers` omitted → the source match's roster. Passing a different roster is the counterfactual lever: replay the same world from the decisive turn with a different model in the losing seat. - `turns` must be greater than the resumed turn; omitted → the scenario's limit. The response is shaped exactly like `POST /bench/matches` (hand each driver only its own token, same as always) plus `resumedFrom`/`resumedTurn`, and those two fields also appear in the match status and its recap. Works for live matches, finished ones, and half-run casualties of a crash — anything that still has a `saves/` directory. ### `GET /health` `{"status":"ok","matches":2,"capstr":"…","engine":"…"}` --- ## Reading the results | field | meaning | |---|---| | `rank` | 1..N among seats that finished. **`0` means DNF** — see below. | | `score` | Freeciv's own civilization score. | | `alive` | whether the civ survived. Every survivor outranks every casualty. | | `elimTurn` | the turn the civ died (absent while it lives). | | `actions` | accepted actions all match. Very low = the model left units idle. | | `illegal` | rejected actions. High = the model is guessing at the API. | | `stalls` | times the seat blew the turn deadline. | | `delegatedPct` | share of unit-turns handed to the engine's AI via `auto_worker`/`auto_explore`. | | `dnf` | the seat was handed to the engine AI; its result is void. | **Rank is the primary signal**, because it self-calibrates: the opponent is another frontier model, so the benchmark gets harder as models improve and never needs re-tuning. The absolute numbers are kept so that a field where *everyone* played badly is visible as such rather than flattering whoever came first. **Rank rewards conflict, deliberately.** The ordering is: survivors above casualties, always; survivors by score; casualties by how long they lasted (`elimTurn`, later is better); DNF unranked at the bottom. And a match can end before its turn limit: **if only one civ is left alive, it wins by conquest on the spot** (the recap shows a `victory` moment). Agents are told all of this, plus that ranking is winner-take-all — a benchmark seat that plays for a safe second is playing the objective wrong. `delegatedPct` deserves attention. Delegation is legal, but a model that hands its army to the engine has opted out of the thing being measured — a high score with a high `delegatedPct` is the engine's result, not the model's. --- ## The turn barrier — the thing to design your harness around The game advances **only when every seat has ended its turn**, and the engine imposes no clock at all. A model may think for as long as it likes and the world simply waits. Consequences: - **Run all six drivers concurrently.** Sequentially, nothing ever advances: each driver would wait for five seats that are not running. - **`/move` never blocks** on the other seats. It applies your actions and returns immediately. Drivers poll `GET …/observe` until `turnOpen` is true again. This is deliberate — holding the request open would couple every driver's socket timeout to every other model's latency. - **The slowest seat sets the pace.** One model taking five minutes a turn holds the other five idle. On a 120-turn arena that is the difference between hours and days. Budget accordingly. ## Hung agents A seat that sends nothing for `TURN_DEADLINE_SECONDS` (default 1200) is **handed to the engine AI and marked DNF**. Its civ keeps playing — a decaying corpse on the map would distort everyone else's game — but the model's result is void and it is excluded from the ranking rather than ranked last. That seat's driver then gets **HTTP 409** from `/move` and `/observe`: ```json {"error":"your seat exceeded the turn deadline and was handed to the engine AI (DNF)", "dnf":true} ``` **409 is terminal.** Treat it as "this seat is over" and stop the driver; do not retry. The scoreboard row is frozen at the moment of hand-off, so a DNF seat still shows how it was doing when it stalled. This is a hang guard, not a speed limit. It exists so one crashed driver cannot freeze a six-hour match. **Retrying a hung or failed model call is your job, not meerkat's.** A provider timeout, a rate limit, a malformed response — handle those in the driver. If you let them reach the deadline, you lose the seat. --- ## Practical notes - **Development runs on a cheap six-seat arena**, not a smaller one. Scale the roster, not the seat count. A cheap arena costs roughly an order of magnitude less than a frontier one and exercises identical concurrency, barrier, fog and recording behaviour. - **A model that never sends `end_turn` wedges all six seats** until the deadline. Have your driver append `end_turn` if the model omits it. - **Matches are expensive to lose — but no longer lost.** The engine snapshots every turn to `/data/matches/{id}/saves/`, and `POST /bench/matches/{id}/resume` boots a new match from any of them. (Freeciv's `/load` is pregame-only, which is why a resume is a new match and a fresh process rather than a rewind of the old one.) - **Artifacts** land in `/data/matches/{id}/`: `media/` holds the world replay and one fogged MP4 per seat, `saves/` the raw per-turn PNG frames and engine savegames, and `recap.json` the full record. All rendered automatically when the match passes its turn limit — you do not need to poll for it, and it happens even if every driver has already disconnected. - **meerkat does not report anywhere.** It runs games and exposes results; shipping them to an experiment tracker is the harness's job. `recap` gives you everything you need for that: per-turn history for metric series, final standings for a summary, and media paths for artifacts. ## Minimal harness ```python m = post("/bench/matches", {"scenario": "arena-short", "challengers": models}) threads = [start_driver(s["token"], models[i]) for i, s in enumerate(m["seats"])] while not get(f"/bench/matches/{m['match']}")["done"]: sleep(5) print(get(f"/bench/matches/{m['match']}")["rank"]) delete(f"/bench/matches/{m['match']}") ``` A complete reference harness — including a free scripted policy you can put in any seat as a baseline — is `drivers/arena.py` in the meerkat repo.