A system is not real-time because it is fast. It is real-time because it is on time, every time, and because you can prove what happens on the run where it is not. This week you take the loop you built and ask the only questions that matter in production: what is the deadline, what happens when it is missed, and can you replay the miss frame for frame.
Four different numbers, one word. Getting them apart is most of this week, because every mitigation you will build depends on which one is hurting you.
Latency is how old your information is when you act on it. Period is how often you get to act. Jitter is how much the period and the latency vary from one cycle to the next. The deadline is the instant after which the answer is worthless — and it is a property of the world, not of your code.
The distinction that surprises people: jitter is worse than latency. A constant 140 ms delay is a known quantity, and you can compensate for it — predict the state forward, lower the gain, design for it. A delay that is 20 ms on most cycles and 180 ms on one in fifty cannot be compensated for, because on any given cycle you do not know which one you are in. You end up designing for the worst case anyway, and paying the average case for nothing.
Hence worst-case thinking. A hard real-time system is characterised by its worst execution time, not its typical one: WCET, p99.9, the maximum you have ever observed plus a margin. The question is never “how fast is it?” but “what is the slowest it can be, and what happens then?”
# three kinds of real-time, and they are engineering choices hard → a missed deadline is a failure # steering, braking, airbags firm → a late answer is worthless, not fatal # a video frame, a quote soft → a late answer is degraded but useful # a dashboard, a log # the honest way to model a slow loop on a fast world (rtsim) sim.run(controller, control_every=sim.steps_for(0.05)) # -> the controller is polled every 50 ms and its command is HELD # in between: a zero-order hold, not a faster world
rtsim refuses to let you use the wall clock as dt. Lateness is modelled, not
absorbed.| name | what it measures | the fix |
|---|---|---|
| latency | age of your input | predict forward; move the compute closer |
| period | how often you decide | cheaper stages; a faster loop |
| jitter | variance of both | remove allocation, GC, locks, I/O from the path |
| deadline | when the answer dies | you cannot: the world sets it |
await on something unbounded. Every one of them is invisible in a mean and obvious in a
p99.9.The same car, the same gains, the same track as week 8. The only thing that changes is when the controller gets to think — and how reliably. Between decisions the last command is held, which is exactly what a real zero-order-hold loop does.
Two mechanisms carry almost all of the safety engineering in this course: a monitor that notices you have stopped being real-time, and a log that lets you prove afterwards exactly what happened.
A watchdog is a countdown that something healthy has to keep resetting. If the heartbeat stops — the perception process died, the model call hung, the queue starved — the countdown expires and the system moves to a defined state. The word that matters is defined: the failure mode has to be something you chose in advance and tested, not whatever the code happens to do when its inputs go stale.
Kill the perception stage and watch four different answers to the same failure. Move the failure onto the straight and then into the corner — the ranking of the policies changes, which is the point: a safe state is a property of the situation, not of the code.
Now the part that makes a post-mortem possible at all. A run is deterministic if replaying the same inputs, in the same order, with the same seed, produces the same outputs bit for bit. Determinism is not a nice property; it is the difference between “something went wrong around 14:32” and “at step 743 the lane offset read −0.4 m and here is why”.
Record a run to a log, replay it step by step, then change exactly one input at one step and diff the two trajectories. The first divergent step is the thing your post-mortem is about.
# what has to be in the bundle for a replay to prove anything {"schema": "rtsim.run/1", "seed": 0, # every random draw came from here "dt": 0.02, "control_every": 1, # the clock, not the wall clock "track": "oval:60,25", # the world, by value not by reference "gains": {"kp": 0.8, "kd": 1.1, "ki": 0.0}, "noise": {"cross_track": 0.0}, "latency": {"sensor": 0}, "git": "9f2c1ab", # the code is an input too "rows": [[step, t, cte, he, v, steer], ...]} # Obs.as_row() # and the test that makes it a proof rather than a story assert digest(replay(bundle)) == bundle["digest"]
Filtering, fusing, retrying, and offloading to something bigger are all real improvements to accuracy — and every one of them is paid for in milliseconds, out of the same budget the deadline lives in.
Take the most common one. Your lane reading is noisy, so you filter it. A complementary filter blends the
slow, absolute measurement with a fast prediction from the model you already have:
x̂ = α·measured + (1−α)·(x̂prev + predicted change).
It is a Kalman filter with the gain frozen instead of computed, it costs three multiplications, and it works.
But α is a dial between two failures: turn it down and the estimate is smooth and late; turn it
up and it is prompt and noisy. Late is a phase lag, and you saw last week what a phase lag does.
One noisy lane reading, one complementary filter. Read both numbers: the RMSE tells you how much noise you removed, the lag tells you what it cost — and the lag is charged to the same budget as everything else.
The same arithmetic settles the argument everyone has about model size. A bigger model somewhere else is not an improvement if the round trip does not fit inside the deadline — and a network round trip is not a number you control, it is a distribution with a long tail, which is week 5's lesson arriving with a steering wheel attached.
The cloud model is more accurate and cheaper per FLOP. Put both against the loop deadline and the choice stops being a preference.
Each of these is a real post-mortem that could not be written. Open them.
“Average loop time 8 ms, well inside 20.” Meanwhile p99.9 is 240 ms because of a garbage collection that happens once a minute, and once a minute the car drives two metres with its eyes shut. The mean was never the wrong number to compute — it was the wrong number to promise.
A hard real-time claim is a claim about the maximum. If you cannot bound it, you do not have a hard real-time system; you have a fast one, which is a different product.
Fix: report p50 / p99 / p99.9 / max and the deadline-miss count, alarm on the miss count rather than on the mean, and state the consequence of a miss in the specification.
The log has outputs but not inputs. Or it has inputs but no seed, so every replay draws different
noise. Or the loop calls time.time(), so the timing is different every run. Or the order of a
dict or a set decides which detection wins, and that order changed between runs. The result is a log that
documents that something happened without letting you reproduce it — which is a diary, not
telemetry.
Fix: log every input, the seed, the config, the code version, and a digest of the output. Then write the test that replays the bundle and asserts the digest. If that test does not exist, your determinism is a belief.
logger.error("perception stale!") and then the loop carries on using the stale value. The
monitor is technically working — it noticed — and the system is exactly as unsafe as it would
have been without it, plus a log line that will be very embarrassing in the post-mortem.
Its cousin: a watchdog whose recovery path needs the thing that died. Reconnecting requires the network that is down, the safe state needs a model call, the alarm allocates memory during a memory failure.
Fix: the monitor must change what the actuator does, along a path that has no dependency on the failed component — and you must test it by killing the component, on purpose, in CI.
An exception propagates out of the control task and the process exits. In a web service that is correct and admirable: crash, restart, serve again. In a moving vehicle it means the last command is held by hardware forever, or the actuator drops to zero at whatever angle it happened to be at. “Restart the process” is not a safe state; it is an absence of one.
Fix: decide, in writing, what the safe state is for each subsystem — and make the degraded mode reachable, tested and boring. Fail fast in the tooling, fail safe in the loop.
Every one of these is a jitter source that will not show up on your laptop with one client and will show up in the demo. A fresh numpy array per frame, an f-string built for a log line that is thrown away, a blocking disk write, a lock another task holds while it does I/O, an unbounded queue that grows until the allocator has to work for it.
Fix: pre-allocate and reuse buffers; log to a bounded ring buffer and drain it off the hot
path; never hold a lock across an await or an I/O call; bound every queue. Then measure the
p99.9 again and see what you actually bought.
Instant feedback, no grading. Pick the answer you would defend in a design review.
Q1Loop A has a constant 100 ms latency. Loop B averages 20 ms but ranges from 10 to 150 ms. Which is harder to control, and why?
Q2You want a crash to be replayable frame for frame. What is the minimum the bundle has to contain?
Q3The lane reading is noisy, so you set the complementary filter's α to 0.05. RMSE drops nicely and the car starts weaving. What did you do?
dt, one seed, every input logged, a digest of
the output.make install, and put the replay harness beside your loop.