Challenges: design a system to meet a stated requirement

[xevrion]

Two people have now asked for this independently, most recently someone who taught cloud development:

you could have like 2 modes or "hey the client has this amount of users per second + needs 3 regions, design and deploy a system under these requirements"

That framing is better than what was in TODO.md, because it is a brief rather than a score. The sandbox teaches whoever already knows what to try. A brief teaches everyone else, which is the audience this tool is actually for.

What already exists

Most of the pieces are in place, which is why this is worth doing now rather than later.

• region is a node kind with real failover, a failover window and a region-down failure reason, so "needs 3 regions" is a topology assertion rather than new engine work.
• The cost model in src/content/vendors/cost.ts gives a design a dollar figure from cited vendor rates, so a budget constraint is readable today.
• SystemStats carries goodputRps, errorRate, p50, p95 and p99, which covers every performance condition anyone has asked for.
• Traffic patterns (ramp, spike, diurnal) are in the engine, so a brief can say "survive the spike" and not only "hold steady load".
• encodeTopology puts a design in the URL, so a challenge could be handed to a class as a link.

So a challenge is roughly: a starting topology, a brief in words, and a pass condition the engine can evaluate against a snapshot.

What needs deciding first

This is tagged discussion rather than good first issue because the design decisions matter more than the code, and getting them wrong makes the feature feel like a quiz rather than a lesson.

What counts as passing. Sustained p99 under the threshold for a whole run, or never exceeding it once? The second is stricter and closer to a real SLO, but a single unlucky spike failing an otherwise sound design teaches the wrong lesson.

Whether the budget is a gate or a score. "Under $200 a month" is pass or fail. "Cheapest passing design wins" turns it into something you replay, which is closer to what "make it a game" meant.

Whether a failed run explains itself. A challenge that says only "failed" is worse than no challenge. It should be able to say which condition broke and point at the component responsible, and the request tracer already knows enough to do that.

How many, and where they live. A handful of good ones beats twenty thin ones. They could be a new field on Preset rather than a separate concept, since a challenge is a preset plus a goal.

Not deciding here

Whether this is a separate mode or a layer over the existing examples. I lean towards the latter, since every one of the 23 examples already teaches a specific failure and most would make a reasonable brief with a sentence and a threshold added.

Comments welcome, particularly from anyone who has taught this material and knows which briefs actually land with students.

[xevrion]

Worth saying before anyone starts: this does not have to be big, and I would rather it were not on the first pass.

The expensive part of "add lots of challenges" is authoring scenarios, and there are already 23 of those. Each one has a topology and a description of the failure it teaches. A challenge on top of an existing preset is a brief in words plus a condition the engine checks, so most of the work is done.

I prototyped the whole thing against retry-storm to see whether it is actually fun before committing to a design. Brief: hold 150 rps with p99 under 200ms and errors under 1%. The only thing the student changes is the database capacity.

db capacity 4    p99=   0ms  errors= 99.3%  fail
db capacity 6    p99=   0ms  errors= 97.8%  fail
db capacity 8    p99= 121ms  errors=  0.0%  PASS
db capacity 10   p99=  89ms  errors=  0.0%  PASS

Two things that came out of that and I think should shape the design.

The interesting part is the cliff between 6 and 8, not the passing. Going from total collapse to comfortable in one step is the moment worth building around, and it is already there in the engine without any new work.

The failure mode teaches on its own. At capacity 4 the p99 reads 0ms next to 99% errors, because nothing completes so there is no latency to measure. A student who notices that has learned something real about what a percentile means on a dead system, and no wording in a brief would have taught it as well.

So the smallest useful version is one challenge on one existing preset, shipped to find out whether people replay it. If they do, the second is cheap. If they do not, we learned that for the price of a day rather than a fortnight.

The thing I would not skip even in the smallest version is the failure explanation. "Failed" on its own is worse than no challenge; it has to say which condition broke.

[xevrion]

A budget ceiling came up, and it is a better idea than I first credited:

some mechanism for a budget? maybe not like the real prices of a cloud provider but some kind of in-game currency to not just "hey autoscale to hell" so the challenges have a ceiling

I went to check how bad the no-ceiling version actually is, on the same retry-storm brief as above, and it is worse than "the answer is boring":

cap=8  inst=1   p99=121ms   errors=0.0%
cap=16 inst=4   p99=120ms   errors=0.0%

Eight times the hardware. One millisecond better. The database stopped being the bottleneck somewhere before that and the rest is spent on nothing.

So a challenge with no ceiling does not merely have a dull answer, it teaches a false one. A student who scales until it passes concludes that more hardware fixes things, which is the opposite of what every example here exists to show. The ceiling is what forces the actual lesson: find the thing that is actually limiting you, rather than paying to avoid finding out.

On real prices versus in-game currency, I lean towards the real cost model that already exists, for two reasons.

The numbers are cited. Every rate in src/content/vendors/ carries a source and the date it was read, and "your design costs $340 a month" lands differently from "your design costs 340 coins". Part of what this tool is for is that the numbers are true.

An invented currency has to be balanced, and badly balanced game money is worse than none. Real prices are already balanced by the market.

The catch is that cost is only defined once a component has a vendor size chosen, and vendor mode is off by default with generic selected. So a budget challenge either has to require vendor mode, or fall back to something coarser like total instance count. Requiring vendor mode seems reasonable for a challenge specifically about spending, and it is one more thing the challenge brief can state.

Worth noting this also makes the third question above concrete. If the budget is real money, "cheapest passing design" is a genuinely interesting scoreboard, and it is the version closest to what the job actually is.

[xevrion]

Two more ideas came up while building the first version, worth writing down before they get lost.

Briefs that start from a blank canvas. The four in this first pass all start from a preset that is already failing, so the task is "work out why this breaks and fix it". That is deliberately the easier half. The original framing was "design and deploy a system under these requirements", which means starting from nothing, and that is a genuinely different and harder exercise. Worth having as a second kind rather than a replacement: diagnosing a broken system and designing one from scratch teach different things, and the first is a much gentler place to start.

Submitting a passed challenge, the way an assignment is handed in. I would not build this as it sounds. Submissions imply accounts, a backend, and somewhere to store student work, which is an LMS rather than a simulator, and "everything runs in your browser, nothing leaves the page" is worth more than the feature.

The need underneath it is real though, and there is already a cheap answer: buildShareUrl puts an entire design in a URL. A student who passes copies a link; whoever set the work opens it and sees both the design and whether it still passes, because the verdict is computed from the design rather than stored beside it. No account, no server, nothing to trust.

What that would need is small: the share link carrying which challenge is being attempted, so opening it restores the brief as well as the topology, and something in the panel that offers the link once the goals are met. That is worth doing. A submissions system is not.