WEEK 10

Season Finale

Ten weeks ago you had a Python file. Today you have a distributed system with a deadline, an intelligence module inside its hot path, telemetry that explains it, and a scoreboard that does not care how clever you were — only whether the thing stayed up and earned its return without betting the account. This is the week you demonstrate that, and then explain the one time it broke.

Project P3 · demo, runbook, tagged repo, post-mortem Live scored session Then the final exam — closed book, in session
Part 1

1What did we build?

One event bus, four core services, and five intelligence modules that plug into it as event sources. Every arrow is a typed message; every box has a latency budget and a defined way to fail.

The shape has not changed since week 3. What changed is what hangs off it: week 5 added a second venue, week 6 a detector reading the session tape, week 7 a perception service, week 8 a closed-loop controller, week 9 a watchdog and a replay recorder. None of them required rewriting the bus — which is the entire argument for the plugin registry, and the reason your bot from week 4 still runs today.

Widget 1 · the season, end to end

Pick a component. The diagram highlights it and its edges, and the panel tells you what it owns, which week built it, the message it puts on the wire, its budget, and how it fails. Arrow keys move along the row.

# the seam that made all five modules possible: one registry, three verbs
arena.register_security("BTC-USD", price_fn=crypto_feed)      # week 5
arena.register_strategy("my_bot",   signal_fn=my_strategy)     # week 4
arena.register_shock("fed_hike",   apply_fn=my_shock_fn)      # week 3

# and the rule that made the bus survive ten weeks of change
await ws.send(msg.model_dump_json())    # a Pydantic model, never a raw dict
msg = OrderAck.model_validate_json(raw) # a model class, never manual JSON
Everything is an event. A tick, a fill, an alert, a detection, a heartbeat, a deadline miss. Once they all travel as typed messages on one bus, adding a producer is a day's work and adding a consumer is an afternoon — and a post-mortem is a query rather than an archaeology project.
Part 2

2How does it survive the demo?

Because you already broke it, on purpose, in private. Five failure modes cover almost everything that goes wrong in a live session, and each has a boring, specific, testable answer.

The naive bot is not badly written — it is written for the happy path, which is the same thing. It assumes a message that arrives is fresh, an order that is sent is filled, a socket that was open is still open, and an event that arrives once arrives exactly once. Every one of those assumptions is false in a live session, and each one has a cheap fix that costs about five lines.

Widget 2 · the failure-mode drill

Pick a failure and run it down two lanes at once. Same market, same strategy, same capital — the only difference is whether the bot expected the failure.

readiness —

Naive bot — written for the happy path

orders sent
—
bad fills
—
P&L damage
—
recovered
—

Robust bot — written for this session

orders sent
—
bad fills
—
P&L damage
—
recovered
—

  
Part 3

3Why does it matter? Because the score is risk-adjusted

Four teams, one cumulative return, four different rankings. The season score is published up front for exactly this reason: ranking on final net worth rewards one lucky levered bet, and that is not the behaviour a course about engineering wants to select for.

Widget 3 · same P&L, different rank

Every team below finishes the season at the same cumulative return. Move the volatility and the drawdown and watch the ranking rearrange itself — then read the formula underneath and see exactly which term did it.


        
Why the vol floor exists. Idle cash earns a tiny interest every tick, so a do-nothing book has a small positive drift and a volatility of pure floating-point noise. An earlier version of this formula divided one by the other and scored “park the cash and go home” about a billion times a real trader — the exact opposite of the intent. Scoring the return above the risk-free drift makes idle cash score about zero; flooring the denominator stops a near-constant book from exploding; the cap bounds whatever is left. Watch the idle team's score as you move the sliders: it does not move, because there is nothing there to move.

What is actually being graded

P&L is the integration test, not the grade. A well-engineered system that loses money politely beats spaghetti that got lucky.

artefactwhat it proves
tagged repothe demo and the code are the same thing
runbooksomeone else can start it
latency histogramyou measured tails, not averages
replay bundlethe failure is reproducible
risk limit that firedthe limit is real, not decorative
post-mortemyou understand your own system
Ranked within role. Traders against traders, brokers against brokers, exchanges on fee revenue. Comparing a market maker's P&L with a venue's fee take would be comparing two different jobs.

The post-mortem, in five lines

  • What happened, in one sentence a stranger could follow.
  • The telemetry that shows it — a chart, a log excerpt, a replay digest. Not a memory.
  • Why the system allowed it: the missing bound, the untested path, the assumption. Never a person.
  • What you changed, and the test that now fails without the change.
  • What you deliberately did not change, and why. This is the line that shows judgement.
Blameless is not vague. “Someone pushed a bad config” is blame and it is also lazy. “Config changes were unversioned and applied without a restart, so the running process and the file disagreed for 40 minutes” names the same event and points at a fix.

Widget 4 · demo-day readiness

Twelve items, weighted by how badly each one hurts when it is missing. Tick what is genuinely true right now — not what you intend to do on the morning.

readiness
—
items done
—
worst gap
—
What a demo-day failure actually looks like. Not a crash — a laptop that cannot reach the venue because the room's wifi blocks the port, a bot that needs an environment variable only the author knows, a chart that takes ninety seconds to render, or a five-minute explanation of why the number on screen is not the number in the report. All four are runbook failures, and all four are free to prevent the night before.

Widget 5 · the course in one loop

Six domains, ten weeks, one loop. Pick a domain and every stage relabels itself. The shape never changes; only the budget does — and the budget is what makes it engineering rather than analysis.

Part 4

4Five ways to lose a demo you had already won

None of these is a bug in your code. Open them.

A demo that only works on your laptop

No runbook, no one-command start, three terminals in a specific order, an environment variable only you know about, and a live network dependency in a room whose wifi blocks the port. The system works; the demo does not; and from the audience those are indistinguishable.

Fix: one command that starts everything, a runbook a stranger can follow, and a rehearsal from a cold boot with the network unplugged. If the demo needs the internet, record a tape and replay it.

Optimising P&L instead of the system

Two days before the demo, the temptation is to turn the size up. It sometimes works, and when it works it is indistinguishable from having been right, which is precisely why it is not being graded. The scoreboard divides by volatility and penalises drawdown, so the levered bet is punished twice even when it pays.

Fix: spend those two days on the runbook, the replay bundle and the histogram. They are worth more marks and they are the parts that transfer to the job.

A post-mortem with a name in it, or with no telemetry

“Sam pushed a broken config.” That sentence ends the investigation exactly where it should begin: why could an unversioned config reach a running process, and why did nothing notice for forty minutes? The mirror-image failure is the blameless post-mortem with no evidence — a paragraph of recollection, no chart, no log, no replay.

Fix: name mechanisms, not people; attach the telemetry that proves the sequence; and add the test that would now fail. A post-mortem without a new test is a story.

An unreproducible final state

No tag, uncommitted changes on the demo machine, a database that was hand-edited during the session, a config that lives in someone's shell history. Six weeks later nobody — including you — can say what was actually running when the numbers you reported were produced.

Fix: tag the commit you demo, commit the config with it, export the season state, and put the digest of the run in the report. The claim “these numbers came from this tag” should be checkable.

Averages on the slide, tails in the room

“Mean decision latency 4 ms.” Then, live, a p99.9 of 300 ms produces a visible stall and the whole presentation is now about the number you did not put on the slide. It is the same mistake as week 7's frame budget and week 9's deadline, arriving one last time with an audience.

Fix: put p50, p99 and max on the slide, with the miss count. It reads as confidence, it is the honest measurement, and it removes the only question you cannot answer.

Part 5

5Check yourself

Instant feedback, no grading. The last three questions of the course.

Q1Two teams finish the season up exactly 6 %. One did it with 0.3 % per-snapshot volatility and a 2 % drawdown; the other with 3 % volatility and a 30 % drawdown. Who ranks higher?

Q2Mid-session your bot receives the same fill message twice. What is the correct handling?

Q3What makes a post-mortem credible to someone who was not there?

Part 6

6Do the work

In class

  • Live scored session: everything runs at once, and the leaderboard is risk-adjusted.
  • Ten-minute demo per team: the system, one measurement, one failure survived.
  • Blameless post-mortems in the room, with the telemetry on screen.
  • Then the final exam — closed book, in session.

Before the demo

  • P3 — tagged repo, runbook, demo, and a post-mortem of one real failure with the telemetry that explains it.
  • Rehearse from a cold boot, on the room's network, with a stranger reading your runbook.
  • Put p50 / p99 / max and the deadline-miss count on the slide.
  • Decide the one sentence you want the room to remember. Say it first, then prove it.
Code
Get the code
The starter repo is public. Use it as a template, clone it, make install — and keep it after the course: it is a working distributed system.
Map
Course home
The ten-week map, from one Python file to this.
Again
Week 1 · start again →
Re-read week 1 now. The UML you drew before you knew what a deadline was will read very differently.
← Week 9 · Hard Real-Time & Safety Week 1 · start again →