Balancing a Game Nobody Has Played Yet: Deterministic Autoplay, and the One Metric I Got Wrong

Balancing a game usually works like this: you ship it, you wait, you watch a dashboard fill up with player deaths, and then you nudge a number and do it all again. That loop needs players. I did not have players.

I had a tower-defense game I built in Flutter, an unreleased one, and a very specific question: is wave 20 too hard, or is it just hard enough that quitting feels like my fault instead of the game's? The honest answer to "should I make the frost nova cheaper?" is "ship it and find out." But shipping to find out means shipping something I already suspect is wrong, waiting a week for enough runs to matter, and then discovering the change also broke the thing next to it. That is a slow, embarrassing way to learn what a computer could have told me before I shipped anything at all.

So instead of waiting for players, I built a bot that plays the game a few hundred times before breakfast.

This is the story of that harness, the numbers it gave me, and the one time it gave me a number I completely misread. That last part is the interesting part, so I'm burying it in the middle like a responsible adult.

The problem: you can't tune a curve you can't see

Difficulty in this game is a curve, not a number. Enemy HP grows wave over wave, gold trickles in, and whether you survive to wave 25 depends on a dozen interacting decisions — which tower you draw, when you merge, what you sell to make room. Change one constant and the whole curve shifts, sometimes in the wrong place.

The thing about a curve is that a single playthrough tells you almost nothing about it. I die on wave 19. Is that the balance, or did I just draw badly? I play again, reach wave 24, feel great about my design. I have now measured my own luck twice and called it game balance.

To see a curve you need the distribution, not the anecdote. And to get a distribution before you have players, you need someone — or something — willing to play the same game two hundred times without getting bored or getting better. Humans are bad at both. I am, at least.

The harness: determinism is the whole trick

The one design decision that made all of this possible predates the balance work entirely. The game's rules live in a plain Dart simulation — call it GameSim — that imports nothing from Flutter. No widgets, no rendering, no BuildContext. The pretty part, the towers and the particles, is a CustomPaint layer that reads the sim and draws it. The sim doesn't know the view exists.

That separation buys one thing that matters more than clean architecture ever does on its own: determinism. Same seed in, same run out. Wave for wave, kill for kill, identically, every time. A run is a pure function of its seed and the rules. Nothing about the frame rate, the device, or the phase of the moon leaks in.

Determinism is what makes a run reproducible, and reproducibility is what makes a bot useful. If seed 47 produced a different game each time, a bot's results would be noise. Because it doesn't, a bot's results are a measurement.

So there's a headless harness — tool/autoplay.dart — that wires a scripted policy to the sim and plays it out:

dart run tool/autoplay.dart --seeds 200

No Flutter engine, no window, no CustomPaint. Just the sim, ticked at a fixed 1/60 substep with the bot acting on a half-second cadence, exactly like a player who taps every 0.5 seconds and never gets tired. It takes a --policy <name> to pick one bot instead of the whole roster, and a --mutator <name> to test the daily-challenge rule variants.

The bots are deliberately dumb. The default one, greedy, does the obvious profitable thing at every step: draw when it can afford to, merge whenever a merge is legal, sell the weakest tower when the board is full and something has to give. It is not trying to be a good player. It is trying to be a consistent one, because consistency is what turns two hundred runs into a curve instead of a cloud.

Reading the numbers: medians, not heroics

Run the harness and each policy prints a line like this:

greedy            wave p25/p50/p75: 21/22/23 (max ...) | spread 2 | kills ... | merges ...

That is a 100-seed report for one policy. The number I care about is the middle one, p50 — the median death wave. Half the runs died before it, half after. The median is the difficulty. The p25 and p75 bracket the typical spread. In this run greedy sits right around wave 22, which is where I wanted it: hard, survivable, not a wall.

But a single median only tells you how hard. It can't tell you why, and it can't tell you whether a change you made did the thing you intended. For that you need two runs that differ in exactly one thing.

This is the core pattern, and it's almost aggressively simple: hold the seed set fixed, change one decision, compare the medians. Everything that isn't the decision under test — the enemy waves, the gold curve, the RNG — is held identical across both bots because they play the same seeds. The only thing that moved is the one choice you're evaluating. So the difference between their medians is caused by that choice and nothing else.

The draft pick is the clean example. There are two bots, greedyDraft and greedyWorstDraft, that are byte-for-byte identical to greedy except for which offer they take when the game freezes and forces a boss draft — the best possible merge-material pick versus the worst. The gap between their medians is a direct measurement of what that single pick is worth. No player survey required. Just two bots, one difference, the same shared seeds.

Here's the honest wrinkle that I like precisely because it's unflattering: over the pinned 200-seed run, greedy and greedyDraft both land at a median death wave of 23. Identical. The best draft pick moved the median by zero. Not because the draft is worthless, but because greedy already saturates — it draws and merges so aggressively that by the time the draft appears, a better offer changes nothing it hadn't already achieved. The measurement didn't fail. It told me the truth: for a maxed-out player, this particular choice is inert. That's a real balance finding, and I only have it because I compared two bots that differed in one place.

That is the harness working as intended. Now the part where I held it wrong.

The metric I got wrong: spread is not agency

Somewhere earlier, I had convinced myself of a very clever idea.

"Agency" — the sense that the player's choices actually matter — is the thing every game designer worries about and nobody can measure. And I thought I'd found a proxy. If I ran the greedy bot across a hundred seeds and its outcomes spread out — some runs dying on wave 18, some on 24 — then surely that spread was evidence that the game has meaningful decisions in it. A wide spread means the game is responsive. A narrow one means it's on rails. Right?

It's a tidy theory. It's also completely wrong, and I want to be precise about why, because the wrongness is subtle enough that I believed it for a while.

The greedy bot has no agency. It follows one fixed rule. It does not weigh options, it does not adapt, it does not make the kind of choice a human makes. When its outcomes spread across seeds, that spread is not the game offering meaningful decisions — it is the leftover randomness of the seed shining through a policy that makes the same move every time. It is seed variance wearing agency's clothes.

I had confused two different things:

• Policy variance — outcomes differ because the agent decided differently. This is what agency looks like in a measurement.
• Seed variance — outcomes differ because the inputs were random while the agent followed one unchanging rule. This is just noise with good PR.

A fixed greedy policy converts seed spread into a measurement of residual RNG. Dressing that up as a claim about player choice is a category error. The bot's spread can tell me the game is random. It cannot tell me the game is interesting.

The audit that finally killed the idea was blunt about it. I ran the balance sweep across seven variants of the economy — genuinely different game shapes — and watched what the greedy bot's numbers did. The median moved from 20 to 31. The game got dramatically harder and easier across those variants; the difficulty curve swung by eleven waves. And the greedy bot's interquartile spread over that entire sweep? It stayed pinned at 2 to 4 waves. Flat. The one number I'd nominated as my agency signal sat almost perfectly still while the thing it was supposed to track lurched across the room.

If a metric doesn't move when the underlying reality moves by eleven waves, it wasn't measuring the reality. It was measuring the bot.

So the spread metric is gone. The replacement is the pattern I'd already been using for the draft pick without noticing it was the answer: to measure whether a choice matters, you don't stare at the noise of one bot — you build two bots that use the choice differently and compare their medians. Policy variance is the signal. Seed variance is, at best, a secondary stability check. And the hard rule I wrote down for myself afterward: a difference between two bot policies is a fact about those two bots. It is not a claim about whether a human's choices matter in general. I don't get to launder one into the other.

For the record, the tuning above still holds. After the frost-nova change, greedy printed 21/22/23; after a later change, 21/22/24. Those numbers say the current build is where I want it right now. They do not permanently establish that the game is balanced, and I've stopped pretending any single run does. That's kind of the whole lesson.

A note on trusting the harness

One reasonable objection: if the balance rides entirely on this simulation, how do I know the simulation is right? Fair. The sim is the most-tested thing in the project — at the pinned commit, flutter test runs 209 passing tests, and coverage of the sim and its rules sits at 95.4% once the genuinely untestable bits (the platform glue, the pixel-pushing painter) are excluded, comfortably over my 90% floor. The harness also asserts sim invariants on every single tick — gold never goes negative, enemies never flood past a cap — so a run that produces a nonsense number crashes instead of quietly lying to me.

(The README still claims the project has 161 tests. It has 209. Documentation ages like milk, and mine is no exception.)

Lessons Learned

1. Determinism is a testing feature, not just a correctness one. Separating the sim from the view was an architecture decision. Getting reproducible, botable balance runs out of it was a free and enormous side benefit. Same seed, same run — build for that early and the measurements come for free later.

2. A distribution beats an anecdote, always. One playthrough measures your luck. Two hundred deterministic runs measure your game. If you're tuning off your own sessions, you are tuning off a sample size of one very biased player.

3. To value a choice, isolate it. Two bots, one difference, the same seeds. The gap between their medians is the value of that difference, cleanly, with nothing else contaminating it. When the gap is zero, that's a finding too.

4. Ask what your metric moves with. The trap wasn't a bad metric, it was an unfalsified one. I never checked whether the greedy spread actually tracked the thing I claimed it measured. When I finally did — eleven waves of median movement, two waves of spread — the metric fell apart in one look. Before you trust a number, make it move.

5. A bot's behavior is a fact about the bot. The most expensive mistake here was generalizing "this fixed policy's outputs spread out" into "the player has agency." A greedy bot cannot tell you what a human's choices are worth. Only a comparison between deliberately different choices can, and even then, the claim stays scoped to what you actually varied.

You don't need players to start balancing a game. You need a deterministic core, a bot dumb enough to be consistent, and the discipline to measure the thing you meant to measure instead of the thing that happened to be handy.

The players can come later. By then, hopefully, the frost nova costs the right amount.

If you've built a simulation-driven game and found a cleaner way to measure agency than "compare medians across policies," I would genuinely like to hear it — the comment section is open, and I've already been wrong once.