One machine that decides now is hard; several machines that must agree on what “now” means is a different problem — and crypto is the case study, because it is distributed by construction and never closes.
Nearly every distributed system you will build is the same object underneath: an append-only log that several machines hold copies of. An exchange's order book is a log of accepted orders. A blockchain is a log of blocks. A Raft cluster is a log with a leader. Get the log right and the application is a fold over it; get it wrong and no amount of application logic saves you.
The reason it is hard is not the appending — it is the agreeing. Machines fail, links drop, and a message that has not arrived is indistinguishable from a message that will never arrive. So a replicated log needs a rule for when an entry is committed: safe to act on, and guaranteed never to be taken back. The usual rule is a quorum — a majority has it, so any future majority must intersect this one and will find it.
One node sequences everything. Cheap, fast, easy to reason about, and the order is the leader's choice. Its weakness is the leader: lose it and you must elect another before you can accept a single write.
Nobody sequences; whoever solves a puzzle proposes the next entry. No election, no membership list, open to strangers. You pay for that with probabilistic commitment: an entry is only probably final, and gets more final with depth.
And a partition forces a choice. When a node cannot reach the quorum, it can refuse to serve — consistent but unavailable — or serve from what it has — available but possibly wrong. That is CAP, and it is not a theoretical curiosity: it is the decision your on-call runbook has to have already made.
Consistency. Every read sees the latest committed write. In a market this is not optional: two venues that disagree about whether your order was accepted will eventually disagree about your position.
Availability. Every request gets an answer. A venue that stops answering during a shock has failed its users just as surely as one that answers wrongly.
Partition tolerance. Not a choice — the network will partition. P is the weather; C and A are what you decide to wear.
Remove the membership list and the leader, and you cannot take a majority vote — anyone can create a thousand identities. Proof of work replaces “one node, one vote” with “one hash, one vote”: to propose the next entry you must find a nonce whose block hash falls below a target. Nobody can fake having done the work, and nobody needs permission to try.
The consequence is that commitment becomes probabilistic. Two miners can solve at the same height, and the chain forks; the rule “follow the heaviest chain” resolves it a block or two later, and whichever branch loses is orphaned along with the transactions only it contained. So “confirmed” is a probability, and the number of confirmations you insist on is a risk decision, priced against how long you are willing to wait.
| Depth | P(attacker wins) | Verdict |
|---|
nonce ‖ prev_hash, counting leading zero bits, in batches on a timer so the page
stays responsive. Expected tries for b bits is 2b, so every extra bit
doubles the work — that is the whole difficulty mechanism. The confirmation table is
(q/p)k: the chance an attacker with q of the hashpower catches up from
k blocks behind.Between two machines there is no such thing as a reliable send. You can retry, and you can deduplicate, and out of those two primitives come the only three guarantees on offer. Choosing among them is not a networking detail — for an order-entry gateway it is the risk model.
Send and forget. No retry, so no duplicate — and no delivery either, if the packet drops. Fine for a market-data tick you will resend in 100 ms. Fatal for “flatten my position”.
Retry until acknowledged. The message definitely arrives — possibly twice, because a lost ack is indistinguishable from a lost request. You end up long 200 when you meant 100.
At-least-once delivery plus an idempotency key: the client stamps a
client_order_id, the venue remembers it, and the second copy is answered from the
first result. Delivery is still at-least-once; the effect is once.
client_order_ids and replays the stored result instead of matching again
— which is why the position error is zero even though the wire still carries duplicates. The
key must be chosen by the client and be stable across retries; a server-generated id or a
timestamp is not an idempotency key.Crypto is the natural laboratory: the same asset trades on many venues at once, all day, with no consolidated tape and no closing bell. If venue B bids above venue A's ask, you appear to be able to buy on A and sell on B for a certain profit. That certainty is an illusion sustained by latency.
By the time your two orders arrive, both books have moved. The edge you measured was the edge that existed one round trip ago, and the trades that reach the market are the ones where it has already gone — you are systematically selected against. Add a clock skew between the venues and you cannot even tell which quote is newer, so “use the fresher book” silently becomes “use the wrong book”. Add a retry and you can end up filled twice on one leg and once on the other, holding an outright position you never wanted.
Each of these is a system that passes its tests on one machine and fails in a way that looks like a market event rather than a bug.
You receive a quote from each venue, each carrying the venue's own timestamp, and you decide which is fresher by comparing them. But the two clocks disagree — by milliseconds if they are well-run, by hundreds if they are not — so the comparison measures skew, not age. Worse, the skew drifts, so a strategy that worked yesterday is systematically wrong today.
The fix. Timestamp on receipt, with your own monotonic clock, and treat venue timestamps as opaque labels for ordering within that venue only. If you must compare across venues, measure the skew explicitly (round-trip probes) and carry it as a known error bar — then check whether your edge survives being wrong by that much.
Your send times out, so you retry — sensible, and the standard advice. But the first request may have arrived and only its ack was lost, so now you own twice the position you sized. The failure is silent: both fills are legitimate, your risk checks pass on each one individually, and you find out from the position report.
The fix. Every order carries a client-generated
client_order_id that is stable across retries, and the venue answers a repeat
with the original outcome. Then reconcile: on reconnect, ask for your open orders and positions and
believe the venue, not your own memory of what you sent.
A crypto deposit appears in the mempool, or one block deep, and your system credits it. Then a reorg orphans that block and the transaction is gone — but the trade you allowed against the credit is not. This is the same mistake as acting on an uncommitted log entry, and it has the same shape: you treated a probabilistic state as a final one.
The fix. Pick a confirmation depth from the attacker model, not from impatience — widget 2 prices it — and make the depth a function of size. Then make the credit reversible in your own ledger, so that if the chain rewrites history your books can follow.
A node stops answering, so you conclude it is dead and promote another. If it was merely unreachable, you now have two leaders, both accepting writes, and the merge will discard some of them. In a market that means acknowledged orders that never existed. The general form: silence is not information, and “dead” and “slow” are the same observation.
The fix. Make promotion require a quorum, so a minority can never elect anyone, and fence the old leader — a term number that the storage layer checks — so its late writes are rejected rather than merged. And decide, in advance and in writing, whether the minority side refuses service or serves stale data.
You log the trades you did, compute the average realised edge, and conclude the strategy works. But the trades you got are the ones nobody faster wanted — the sample is selected against you. The population you needed was every observed opportunity, including the ones that had evaporated by the time your order arrived.
The fix. Log the opportunity, not just the trade: observed edge at decision time, realised edge at fill time, and the difference. That difference, plotted against latency, is your adverse-selection curve, and it tells you whether the answer is a faster link or a wider threshold.
Instant feedback, nothing recorded. Pick the best answer.
Two venues quoting the same crypto pair: measure the arbitrage window, then add 200 ms of latency and partition a link. Then write the CAP analysis of your own venue pair — what you chose, what it costs, and what your runbook does when it happens at 3 a.m.
make install. Labs and deliverables live in your course's own system.