Async I/O & Serialization

Last week you learned what the bytes mean. This week you stop paying full price for them. Two costs sit between the wire and your strategy — the syscall that fetches the bytes and the codec that turns them into numbers — and both of them are usually larger than the strategy itself. You will make one thread service many sockets without ever blocking, and you will make a decode that touches only the fields you trade on.

• Explain readiness versus completion, and why EAGAIN is the normal case rather than an error.
• Choose between select, poll and epoll with the O(n) versus O(ready) argument, and say what edge-triggering obliges you to do.
• Frame a byte stream correctly under partial reads, with a length prefix or a delimiter.
• Beat std::to_string with a twenty-line u64toa, and beat a DOM parser with a string_view.

1What is it? One thread, many sockets, never blocked

A blocking recv() parks your thread in the kernel until bytes arrive. That is fine for one socket and catastrophic for many: a quiet connection freezes the thread that a busy connection needs. The fix is to invert the question — instead of “give me bytes from this socket”, ask “which of my sockets has bytes right now?” and do the reads yourself. That is readiness, and it is what select, poll and epoll answer.

—

The chart plots multiplexing overhead only — the cost of finding out which sockets are ready, before a single recv — at a fixed ready fraction, sweeping the connection count. select and poll hand the kernel the whole interest set on every call and get the whole set back, so their cost grows with registered descriptors — which is exactly the wrong variable. epoll registers once with epoll_ctl and returns only the ready ones, so its cost grows with ready descriptors. At two per cent activity that is a fiftyfold difference in the wrong direction.

Edge versus level triggering

Level-triggered means “this socket is still readable”; edge-triggered means “this socket became readable”. Edge gives you fewer wake-ups and one hard obligation: you must drain the socket until EAGAIN, because nothing will remind you. Step the two side by side.

—

2How does it work? Syscalls, batches, and frames

Once you are non-blocking, the remaining cost is the syscall itself: about 600 ns of round trip into the kernel and back, per call, doing nothing useful. There are exactly three ways to pay less of it — call it less often per message, amortise it over many messages, or stop calling it at all.

a · Busy-poll the socket, or wait to be woken?

—

recvmmsg is the zero-copy-adjacent idea you can actually reach from portable code: one syscall fills an array of mmsghdr structures, so 64 datagrams cost one kernel transition instead of 64. Note what the widget shows about it, though — a batch of 64 only helps if 64 messages are already waiting. If they are not, the call either returns short (no gain) or waits for them (pure added latency). Batching converts throughput headroom into latency, and on a tick path you have the opposite preference.

b · Framing a byte stream under partial reads

TCP does not preserve your send boundaries. A single recv can deliver half a message, two messages, or two and a half. The buffer below receives the same three messages in awkward chunks; watch the framing loop cope, and switch the framing strategy.

—

3Why it matters: the codec is on your critical path

Between the byte arriving and your strategy seeing a number sits the codec, and on the arena's JSON wire it is the single largest item in your tick-to-trade budget. Four ways to turn the same book_snapshot into a best bid and ask. Race them, then read the allocation column, because that is where your p99.9 lives.

—

Look at the p99.9 column rather than the mean. The DOM parser's median is bad and its tail is far worse, because every one of its twenty-two allocations can be the one that walks a free list or asks the kernel for a page. A codec with zero allocations has a tail that looks like its median — that is the whole reason to hunt allocations rather than instructions.

Allocation-free parsing: string_view is the whole trick

// SLOW: every call constructs and destroys a string
std::string get_symbol(const json& j) {
  return j["symbol"].get<std::string>();
}                       // 1 malloc + 1 free + a copy

// FAST: a view into bytes you already have
std::string_view get_symbol(std::string_view frame) {
  auto k = frame.find("\"symbol\":\"");
  if (k == frame.npos) return {};
  k += 10;
  auto e = frame.find('"', k);
  return frame.substr(k, e - k);
}                       // 0 malloc, 0 copy

The other direction: u64toa

Serializing is the half people forget. Every outbound order stamps an id, a quantity and a price into a buffer, and the obvious tools are both terrible: snprintf parses a format string and consults the locale at run time, and std::to_string allocates a std::string and destroys it. Twenty lines beat both. Step the digits.

Skipping the kernel entirely: shared memory

Two processes on one box do not need a network stack to talk. A ring in a shared-memory region — the starter's shm_ring.hpp — is the same SPSC structure from Week 10 with one extra constraint: no pointers, because the region maps at a different address in each process. Compare it against the alternatives.

// Project Phase 4 — a POD ring in a shared region.
// NO pointers: they mean different things in
// each process. Offsets and indices only.
struct ShmRing {
  static constexpr uint32_t CAPACITY = 1024;
  void init();                  // creator, pre-fork
  bool push(uint64_t v);         // producer process
  bool pop(uint64_t& out);       // consumer process
  std::atomic<uint32_t> head, tail;
  uint64_t buf[CAPACITY];
};

Two details that catch everyone. First, std::atomic<uint32_t> is fine across processes only because it is lock-free and address-free on every platform you will use — check is_always_lock_free and do not put a std::mutex in shared memory. Second, the struct must be standard-layout and contain no pointers, no std::vector, no std::string: the region maps at a different virtual address in each process, so a pointer written by one is garbage to the other. Indices are position-independent; addresses are not.

4Five ways this goes wrong

Two of these hang, two of them lie, and one of them corrupts memory. Open them.

5Check yourself

Three questions, instant feedback, nothing recorded.

6Do the work