WEEK 8 · FRI NOV 20 · LIVE BLOCK: SEASON WEEK 8

Machine Learning Signals & Model Ops

A model is just another signal — a number your bot still has to size, risk-manage and route. This week you learn to earn that number honestly: features known at time t, splits that respect time, and a validated artifact that predicts inside your per-tick budget.

Build a feature matrix from prices and the order book without leaking the future into a single column.
Validate walk-forward and recognise the exact moment a backtest score stops being a measurement.
Operate a model: persist it, load it once, predict in on_tick, and watch it drift.

What is it?

Supervised learning, pointed at the future

Learn a mapping f(X) → y from historical examples. In trading, X is everything you know now; y is the piece of the future you are predicting.

Every row of your dataset is one point in time. The features in that row must be computable from data that had already printed at that instant — the last mid, the rolling volatility over the previous twenty ticks, the current book imbalance. The label belongs to the future: the return from t to t+h, or merely its sign.

Two label choices, two different jobs. Classification predicts direction and is robust — you can size by the predicted probability. Regression predicts magnitude, is more informative, and is noisier and harder. Either way the horizon h must match how long you actually intend to hold: a label about the next 30 seconds cannot justify a position you keep for an hour.

The golden rule has one clause. Every feature in X must be knowable at t. Anything else is leakage, and leakage does not look like a bug — it looks like a great model.

# one row per point-in-time observation
X = features known AT time t     # no future data!
y = label about t+h              # the horizon

# classification  -> sign of the next return
y = 1 if ret(t, t+h) > 0 else 0
# regression      -> magnitude of the next return
y = ret(t, t+h)

model.fit(X_train, y_train)
y_hat = model.predict(X_now)     # -> a signal

Why it is dangerous here

low SNR

Three properties of market data that make ML in finance different from ML on images.

Signal-to-noise

~52%

Returns are mostly noise. A model at 52% out-of-sample can be excellent — or pure overfit. You cannot tell by looking at it.

Non-stationarity

drift

The data-generating process changes. Yesterday's edge decays and a model trained on it rots silently, still returning confident numbers.

Feedback & scarcity

n small

Little independent data, and your own orders move the price. Backtests flatter; the live session collects.

Simple first. Every W3–W7 signal is a baseline. ML has to beat it out-of-sample, after costs to earn its place in your bot.

Widget 1

Feature-engineering playground

A synthetic tick series for one symbol. Pick features, set the label horizon, and watch the design matrix assemble. Click a row in the matrix to see which instant it was cut from.

horizon h 5 ticks

Matrix shape

—

usable rows × features

Rows per feature

—

degrees of freedom you are spending

Read the warm-up. vol_20 needs twenty prior returns and the label needs h future ticks, so the first rows and the last h rows are unusable. Dropping them is correct; quietly filling them is leakage. Features are cheap — labels are precious.

How does it work?

Splitting in time is the whole discipline

Fitting the model is one line. Everything that decides whether the number means anything happens in how you cut the data — and in how many times you look.

Random shuffling is the default in every tutorial and it is catastrophic on a time series. Shuffle, and a row from next Tuesday sits in your training set while its neighbour — nearly the same market state — sits in the test set. The model does not learn structure; it learns to recognise its neighbours. Your test score becomes a measurement of how similar adjacent ticks are, which is close to 100% and completely useless.

Widget 2

Leakage detector

The same data, the same model, three ways of cutting it. Watch the reported test accuracy move while the live accuracy does not.

train share 70%

embargo 5 ticks

In-sample

—

accuracy on the training block

Reported test

—

the number you would put in your report

Live

—

what the arena actually pays you for

Why the embargo. A label at time t looks h ticks ahead, so rows within h of the boundary see across the wall even under a clean chronological split. Purging those rows costs you a handful of samples and buys you a number you can trust.

Once the wall is in the right place, one split is still not an evaluation — it is a single sample. Walk-forward (rolling-origin) validation trains on the past, tests on the next unseen block, rolls forward and repeats. Every test block is out-of-sample, which is exactly the situation your bot is in at every tick of the live session: it only ever knows the past.

Widget 3

Walk-forward validation animator

Step the folds forward and collect the out-of-sample score of each test block. Then flip on a regime shift and watch the expanding window — which never forgets — lose to the rolling one.

folds 5

regime shift at fold 3

Mean OOS score

—

the only score worth reporting

Dispersion across folds

—

a good mean with huge spread is not an edge

Roll the folds forward to collect out-of-sample scores.

In AlgoArena. tests/sim_session.py runs the whole arena in one process with no network, so a fold is one call: fit on history[:train_end], then run the bot over the test block only and read the out-of-sample P&L. Same bot object you will ship — validate it honestly, then do not touch it.

Why does it matter in trading?

Accuracy is a vanity metric. Net, risk-adjusted P&L is the score.

A 55% classifier that flips every tick pays the spread endlessly and finishes the season below idle cash. A 52% classifier that trades forty times pays for itself.

The arena settles this argument with numbers. Every fill pays a taker fee or earns a maker rebate, every crossing of the spread costs half of it, and the season leaderboard ranks risk-adjusted return — not raw P&L. So the question a model has to answer is never "is it accurate?" but "does its edge per trade survive the cost of the trades it wants to make?"

Widget 5

Does the edge survive the costs?

Your model's statistical edge on the left; the arena's toll booth on the right. The equity curve is the same signal, before and after costs.

hit rate 53.0%

avg win / avg loss 1.00

trades / session 60

cost / trade 3.0 bp

Gross edge / trade

—

expectancy before fees

Cost drag

—

trades × cost, per session

Net session P&L

—

what the ledger records

Sharpe-like

—

mean / stdev of trade returns

Turnover is a design parameter, not an accident. Raise the confidence threshold in on_tick and you trade less, pay less, and keep more of a smaller edge. Slower, surer signals win seasons in this arena because the score divides by volatility.

Train offline · load once · predict in `on_tick`

The split that keeps you inside the latency budget: heavy work outside the loop, only predict() inside it.

# team/trader.py
import joblib
from arena import Trader, Signal

class MLTrader(Trader):
    def __init__(self):
        self.model = joblib.load('model.pkl')   # load ONCE

    def on_tick(self, market, portfolio):
        for sym in market.symbols():
            px = market.prices(sym)
            if len(px) < 21: continue
            X = features(px, market, sym)        # same fn as offline
            p = self.model.predict_proba([X])[0][1]
            if p > 0.60 and portfolio.can_buy(sym, 5, market.mid_price(sym)):
                return Signal(symbol=sym, side='buy', quantity=5,
                              price=market.best_ask(sym), confidence=p)
        return None

The model replaces the signal, not the system

Everything you built in W4–W7 stays exactly where it is.

Layer	W7 bot	W8 bot
Signal	momentum rule	model.predict_proba
Sizing	fixed 5 lots	scaled by confidence
Risk	check_order()	unchanged
Routing	best_ask()	unchanged
Portfolio	cash + positions	unchanged

Two equivalent seams. Register a signal_fn in the plugin registry, or subclass arena.Trader and implement on_tick. The engine calls whichever you registered. This week also fires the season's second IPO — handle it in on_ipo.

Common mistakes

Five ways to produce a beautiful, worthless model

Every one of these has shipped in a real fund. Each expands.

train_test_split(shuffle=True) on a time series

The tutorial default. Adjacent ticks are nearly identical, so a shuffled split puts a row's near-twin on the other side of the wall. The model memorises neighbours and reports accuracy in the 70s and 80s. The tell: a test score far above anything the literature reports for return prediction.

Fix: cut by index, never by permutation — X[:cut] / X[cut:] — and purge the h rows that straddle the boundary.

A feature that quietly contains its own label

A rolling calculation centred instead of trailing, a column derived from the close of the bar you are predicting, a "current session VWAP" that includes the future. Silent and fatal: the model looks brilliant offline and random live. The tell: one feature dominates the importance ranking and the edge collapses the instant you go live.

Fix: for each feature ask "at exactly which timestamp could I have computed this?" Then compute features with the same function offline and inside on_tick — one function, two callers.

Tuning on the test block (data snooping)

Try a hundred configurations, keep the best test score, report it. You have now fitted the test set; its score is a maximum over noise, not a measurement. The published number is inflated by roughly the spread of your hundred attempts. The tell: your best configuration is nowhere near your second best.

Fix: tune on a validation block, look at the test block once, log every experiment you ran, and deflate your Sharpe for the number of trials.

Feature mismatch between training and on_tick

Offline you standardised with the full-sample mean and standard deviation; live you standardise with a rolling one. Offline the imbalance used total depth; live it uses top-of-book. The model receives inputs it has never seen and outputs confident nonsense. The tell: live predictions cluster at one extreme.

Fix: persist the scaler and the feature list with the model, and assert the live feature vector's length and order on the first tick.

A model you cannot run inside a tick

A gradient-boosted forest of 2,000 trees, re-fitted every tick "to stay fresh", loading the pickle inside on_tick. Your bot misses the market it correctly predicted, and the season's rate limits (enforced from Week 8) throttle the backlog. The tell: your fills print at prices two ticks behind your signals.

Fix: load once in __init__, time predict() against the per-tick budget from W6, and prefer a small model you can actually run. A model you can't run live scores nothing.

Check yourself

Three questions

Instant feedback, no grading. Pick the answer you would defend in a demo.

Question 1

Your classifier reports 84% out-of-sample accuracy on next-tick direction for AAPL. What is the single most likely explanation?

C. Next-tick direction on a liquid name is close to a coin flip; the honest range is roughly 50–55%. An 84% is not a good model, it is a diagnostic: something on the other side of the wall got into training. Check the split first, then hunt the feature.

Question 2

You label with a horizon of h = 10 ticks and split chronologically at 70%. Why is an embargo still needed?

B. The wall stops features, not labels. A label is about the future, so rows within h of the cut straddle it. Purge those h rows and the boundary is genuinely sealed — the cost is a handful of samples.

Question 3

Two bots, same walk-forward hit rate of 53%. Bot A trades 30 times a session, bot B trades 350 times. Round-trip cost is 3 bp. Who ranks higher on the season leaderboard, and why?

D. Expectancy is per trade and small; costs are per trade and certain. Multiply both by 350 and the certain term wins. On top of that, exchange/scoring.py ranks return per unit of volatility with a drawdown penalty, so the churn is punished twice.

Do the work

This week's work

Deliverable: a model integration plan plus the walk-forward evaluation checklist — features, label, horizon, folds, one test look, measured predict() latency.

Code

Get the code

Labs and homework live in your course’s LMS; the starter repo is public. Use it as a template, clone, make install.

← Week 7 · Big Data & Distributed Computing All weeks Week 9 · CI/CD, Testing & UI + Season Finale →