Low-Latency Design — SIMD & Kernel Bypass

Week 12 made the wire lean. This week the box itself is the bottleneck: 64-byte cache lines, eight lanes per instruction, a branch predictor you can help or fight, and a kernel network stack you can walk around entirely. Every item here is a measured microsecond, not a style preference.

• Choose AoS or SoA for a given scan and compute, exactly, how many 64-byte lines each one touches.
• Vectorize a hot reduction over alignas(32) data — and name the three things that make the vectorizer give up.
• Price the kernel network path against kernel bypass in µs, then defend one tuning change per checklist line.
• Read the arena's Latency Board and say whether your next p99.9 comes from code or from colocation.

1What is it? Winning back the last microseconds, in three places

By week 13 your algorithm is right and your allocations are gone. What is left is hardware: the memory you touch, the instructions you issue, and the kernel you ask for favours. Those are the three levers, and each one is worth a different number of microseconds — so the order you pull them in matters.

Interactive · array-of-structs vs struct-of-arrays

You are scanning one field of every order — say the price, to recompute an imbalance. Pick a layout and watch how many 64-byte cache lines the scan drags in, and how much of each line you actually use. The grid shows one box per line; violet is useful bytes, grey is bytes you paid for and threw away.

2How it works — lanes, branches and the line you fetch early

Three mechanisms, and they interact. SIMD gives you width; a data-dependent branch takes it straight back; a prefetch hides the latency that neither of them can do anything about. Play with all three before you write an intrinsic by hand.

Interactive · the vector lanes

Step or run the reduction over a price array. Each frame consumes one instruction's worth of data — one lane if you are scalar, eight if you are on AVX2. Then change the loop shape and watch the vectorizer refuse.

Interactive · what a mispredicted branch costs

A correctly predicted branch is nearly free. A miss flushes the pipeline — roughly 15–20 cycles, about 5 ns at 3.4 GHz. Branchless code pays a small fixed cost on every iteration and never flushes. Move the predictability slider and find the crossover.

// one unpredictable branch per level
for (std::size_t i = 0; i < n; ++i) {
  if (px[i] > ref) buy += qty[i];
  else            sell += qty[i];
}

// same answer, no branch to mispredict
for (std::size_t i = 0; i < n; ++i) {
  const auto m = -static_cast<std::int64_t>(px[i] > ref);
  buy  += qty[i] &  m;   // mask is all-ones or 0
  sell += qty[i] & ~m;
}

__builtin_prefetch(&level[i + 4], 0, 3);  // read, keep it: L1
__builtin_prefetch(&tape[i + 16], 0, 0); // read, streaming: don't retain

The week-13 kernel, for real

This compiles as written under clang++ -std=c++20 -O2 on any target: the AVX2 path is guarded, the scalar path is the reference, and both must produce the same number — which is the checksum rule from the lab.

#include <cstddef>
#if defined(__AVX2__)
#include <immintrin.h>
#endif

// Weighted order-book imbalance. obi and w are alignas(32); n % 8 == 0.
float weighted_obi(const float* obi, const float* w, std::size_t n) {
#if defined(__AVX2__)
  __m256 acc = _mm256_setzero_ps();
  for (std::size_t i = 0; i < n; i += 8) {
    __builtin_prefetch(&obi[i + 16], 0, 0);   // the next 64 B line
    __m256 a = _mm256_load_ps(&obi[i]);         // aligned: needs alignas(32)
    __m256 b = _mm256_load_ps(&w[i]);
    acc = _mm256_fmadd_ps(a, b, acc);            // a*b + acc, 8 lanes, 1 instr
  }
  alignas(32) float t[8];
  _mm256_store_ps(t, acc);
  return t[0]+t[1]+t[2]+t[3]+t[4]+t[5]+t[6]+t[7];   // horizontal reduce
#else
  float s = 0.0f;                                 // portable reference path
  for (std::size_t i = 0; i < n; ++i) s += obi[i] * w[i];
  return s;
#endif
}
// release: -O3 -march=native -flto -DNDEBUG -g

Two things to notice. The #if is not defensive clutter — it is what lets one repo build on the grader, on your laptop and on an ARM box, and it gives you a scalar oracle to diff the vector result against. And the horizontal reduce at the end is not free: it is why vectorizing a very short loop can be slower than not bothering.

3Why it matters — the kernel is the biggest single line item

All the SIMD in the world moves a few hundred nanoseconds. The path a packet takes from the wire to your on_book moves microseconds. That is why kernel bypass and core isolation come after the cheap wins but before the expensive ones: they are the largest block on the chart, and the arena's Latency Board reads them off your p99.9.

Interactive · the packet path, stage by stage

Same frame, same NIC, three different paths into your process. Bars are microseconds; the numbers are representative of a modern server NIC under light load and are labelled estimates, not guarantees — the shape is the lesson.

Interactive · the tuning checklist

Eight platform changes, each with an estimated effect on p50 and on p99.9, and each with a cost. Tick them and read the running total — then read the warnings, because several of these are worthless or actively harmful on their own.

What the arena's Latency Board actually measures

4Five ways this goes wrong

Every one of these has been submitted, graded, and argued about. Open them.

5Check yourself

Three questions, instant feedback, nothing recorded.

6Do the work