Back-Testing a Strategy Honestly
Build a bar-by-bar back-tester with next-bar fills and costs, then use pure random walks to watch look-ahead, survivorship and overfitting manufacture results out of noise.
A back-test is an experiment that is easy to rig by accident
A back-test replays past data through a trading rule and reports what would have happened: which orders, which fills, what position, what profit and loss (PnL). It is the standard first test of any systematic rule, and it is easy to get wrong in a way that makes the result look better. Nobody has to cheat. A fill price one bar too early, a universe of assets that quietly drops the failures, a parameter tuned on the data used to judge it: each one turns noise into an apparent edge, and the report looks just as convincing as an honest one.
This lesson builds a small back-tester and then breaks it deliberately, once per bias. The trick that makes every demonstration clear is the data. All prices here are synthetic random walks with zero drift: each price step is drawn independently, with mean zero, from the shared Lcg. On such data no rule that uses only past prices can expect to make money before costs, and after costs it must expect to lose. Any back-test that reports otherwise on average is measuring its own bug.
That also sets the rule for reading every number in this lesson. These results demonstrate method on synthetic data; they say nothing about any real market or any real strategy, and nothing here is investment advice.
Bars, signals, fills and PnL
The input to a back-test here is a series of bars. A bar summarizes one interval (a day, a minute) by four prices: the open (first trade), high, low and close (last trade). The back-tester walks the bars in time order and, at each one, does what a live system would do at that moment:
- Look at the data available so far and compute a signal: the position the rule wants (here
+1lot long,-1lot short). - Turn a change of wanted position into an order, and fill it at a price that was actually available after the decision.
- Charge costs: slippage (paying a little worse than the quoted price, here one tick per lot) and a fee proportional to the notional, the price times the quantity (here 1 basis point, one hundredth of a percent, rounded up).
- Mark the position to market: equity is cash plus position times the current close.
The rule is the textbook moving-average crossover: go long when the average of the last 10 closes is above the average of the last 40, short when below. It is used here because it is simple and needs a tunable parameter pair, not because it means anything.
First, the data. Prices are integer ticks, as everywhere in this course. Each bar opens a random gap of up to 5 ticks away from the previous close and then takes four more random steps of -5 to +5 ticks.
#include <algorithm>
#include <cstdint>
#include <print>
#include <vector>
using Price = std::int64_t; // integer ticks
using Qty = std::int64_t; // whole lots
struct Lcg {
std::uint64_t s;
std::uint32_t next() {
s = s * 6364136223846793005ULL + 1442695040888963407ULL;
return std::uint32_t(s >> 33);
}
int range(int lo, int hi) { return lo + int(next() % std::uint32_t(hi - lo + 1)); }
};
struct Bar { Price open, high, low, close; };
// A synthetic random walk in ticks: a gap of -5..+5 ticks from the previous
// close to the open, then four more steps of -5..+5 ticks inside the bar.
// The steps have mean zero, so no rule that looks only at the past can
// expect to make money on it. That is the point.
std::vector<Bar> make_bars(std::uint64_t seed, int n, Price start = 10'000) {
Lcg rng{seed};
std::vector<Bar> bars;
Price p = start;
for (int i = 0; i < n; ++i) {
p += rng.range(-5, 5);
Bar b{p, p, p, p};
for (int step = 0; step < 4; ++step) {
p += rng.range(-5, 5);
b.high = std::max(b.high, p);
b.low = std::min(b.low, p);
}
b.close = p;
bars.push_back(b);
}
return bars;
}
int main() {
auto bars = make_bars(7, 1000);
std::println("bar open high low close");
for (int i = 0; i < 5; ++i)
std::println("{:>3} {:>6} {:>6} {:>6} {:>6}", i, bars[i].open, bars[i].high,
bars[i].low, bars[i].close);
// Is there anything to predict? Count how often a bar moves in the
// same direction as the bar before it.
int same = 0, pairs = 0;
for (int i = 1; i < int(bars.size()); ++i) {
Price prev = bars[i - 1].close - bars[i - 1].open;
Price cur = bars[i].close - bars[i].open;
if (prev == 0 || cur == 0) continue;
++pairs;
same += (prev > 0) == (cur > 0);
}
auto [lo, hi] = std::ranges::minmax(bars, {}, &Bar::close);
std::println("closes range from {} to {}; last close {}", lo.close, hi.close,
bars.back().close);
std::println("bar moved the same way as the previous bar: {} of {}", same, pairs);
}Press Run to compile and run this example.
bar open high low close
0 10001 10004 10000 10000
1 10001 10001 9989 9989
2 9991 9996 9991 9995
3 9991 9998 9990 9998
4 10000 10005 9999 10005
closes range from 9988 to 10494; last close 10361
bar moved the same way as the previous bar: 456 of 875The direction of one bar tells you nothing about the next: 456 same-direction pairs out of 875 is within about 1.3 standard errors of half. Yet the picture of this series, with the two moving averages and the position the crossover rule holds, looks as if it has trends worth following. Blue bands are long, red bands short; higher prices are higher on the page, since the program flips SVG's downward y-axis.
#include <algorithm>
#include <cstdint>
#include <print>
#include <vector>
using Price = std::int64_t; // integer ticks
using Qty = std::int64_t; // whole lots
struct Lcg {
std::uint64_t s;
std::uint32_t next() {
s = s * 6364136223846793005ULL + 1442695040888963407ULL;
return std::uint32_t(s >> 33);
}
int range(int lo, int hi) { return lo + int(next() % std::uint32_t(hi - lo + 1)); }
};
struct Bar { Price open, high, low, close; };
// A synthetic random walk in ticks: a gap of -5..+5 ticks from the previous
// close to the open, then four more steps of -5..+5 ticks inside the bar.
// The steps have mean zero, so no rule that looks only at the past can
// expect to make money on it. That is the point.
std::vector<Bar> make_bars(std::uint64_t seed, int n, Price start = 10'000) {
Lcg rng{seed};
std::vector<Bar> bars;
Price p = start;
for (int i = 0; i < n; ++i) {
p += rng.range(-5, 5);
Bar b{p, p, p, p};
for (int step = 0; step < 4; ++step) {
p += rng.range(-5, 5);
b.high = std::max(b.high, p);
b.low = std::min(b.low, p);
}
b.close = p;
bars.push_back(b);
}
return bars;
}
// Closes (dark), 10-bar average (blue), 40-bar average (red), and the
// position the crossover rule holds: light blue band = long, light red = short.
int main() {
const int n = 300, fast = 10, slow = 40;
auto bars = make_bars(7, n);
std::vector<double> fma(n, 0), sma(n, 0);
for (int t = 0; t < n; ++t) {
double sf = 0, ss = 0;
for (int i = std::max(0, t - fast + 1); i <= t; ++i) sf += bars[i].close;
for (int i = std::max(0, t - slow + 1); i <= t; ++i) ss += bars[i].close;
fma[t] = sf / fast;
sma[t] = ss / slow;
}
auto [lo_bar, hi_bar] = std::ranges::minmax(bars, {}, &Bar::close);
const double lo = lo_bar.close - 10, hi = hi_bar.close + 10;
auto x = [&](int t) { return 40.0 + 540.0 * t / (n - 1); };
auto y = [&](double p) { return 10.0 + 220.0 * (hi - p) / (hi - lo); }; // y up
std::println(R"(<svg viewBox="0 0 590 250">)");
// One band per run of bars with the same position (long or short).
for (int start = slow - 1; start < n - 1;) {
bool long_pos = fma[start] > sma[start];
int end = start + 1;
while (end < n - 1 && (fma[end] > sma[end]) == long_pos) ++end;
std::println(R"(<rect x="{:.1f}" y="10" width="{:.1f}" height="220" )"
R"(fill="{}" fill-opacity="0.12"/>)", x(start), x(end) - x(start),
long_pos ? "#2563eb" : "#dc2626");
start = end;
}
auto line = [&](auto value, int from, const char* color, double width) {
std::print(R"(<polyline fill="none" stroke="{}" stroke-width="{}" points=")",
color, width);
for (int t = from; t < n; ++t) std::print("{:.1f},{:.1f} ", x(t), y(value(t)));
std::println(R"("/>)");
};
line([&](int t) { return double(bars[t].close); }, 0, "#1f2937", 1);
line([&](int t) { return fma[t]; }, fast - 1, "#2563eb", 1.8);
line([&](int t) { return sma[t]; }, slow - 1, "#dc2626", 1.8);
std::println(R"(<text x="2" y="{:.0f}" font-size="10">{:.0f}</text>)", y(hi) + 4, hi);
std::println(R"(<text x="2" y="{:.0f}" font-size="10">{:.0f}</text>)", y(lo), lo);
std::println(R"(<text x="250" y="246" font-size="11">bar 0 to {}</text>)", n - 1);
std::println("</svg>");
}Press Run to compile and run this example.
Random walks look like that. Long runs and apparent turning points are exactly what a walk with independent steps produces, which is why eyeballing a chart is no evidence of anything.
A back-tester that cannot see the future
The core of the back-tester is one loop over bars. The rule that keeps it honest is in the order of operations: the signal for bar t uses that bar's close, so it is known only at the end of bar t, and the earliest realistic fill is the open of bar t + 1. The function supports two other fill modes, used in the next section to show what goes wrong; Fill::NextOpen is the default.
The moving averages use prefix sums, so each one costs O(1) per bar and a whole back-test is O(n) for n bars. They compare sum_fast * slow with sum_slow * fast, which is the same comparison as the averages but in exact integer arithmetic: no division, no rounding ties. The program ends with an accounting check: the per-bar PnL must add up exactly to the final equity.
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <numeric>
#include <print>
#include <ranges>
#include <vector>
using Price = std::int64_t; // integer ticks
using Qty = std::int64_t; // whole lots
struct Lcg {
std::uint64_t s;
std::uint32_t next() {
s = s * 6364136223846793005ULL + 1442695040888963407ULL;
return std::uint32_t(s >> 33);
}
int range(int lo, int hi) { return lo + int(next() % std::uint32_t(hi - lo + 1)); }
};
struct Bar { Price open, high, low, close; };
// A synthetic random walk in ticks: a gap of -5..+5 ticks from the previous
// close to the open, then four more steps of -5..+5 ticks inside the bar.
// The steps have mean zero, so no rule that looks only at the past can
// expect to make money on it. That is the point.
std::vector<Bar> make_bars(std::uint64_t seed, int n, Price start = 10'000) {
Lcg rng{seed};
std::vector<Bar> bars;
Price p = start;
for (int i = 0; i < n; ++i) {
p += rng.range(-5, 5);
Bar b{p, p, p, p};
for (int step = 0; step < 4; ++step) {
p += rng.range(-5, 5);
b.high = std::max(b.high, p);
b.low = std::min(b.low, p);
}
b.close = p;
bars.push_back(b);
}
return bars;
}
// Simple moving averages from prefix sums: O(1) per query after O(n) setup.
struct Sma {
std::vector<Price> prefix{0};
explicit Sma(const std::vector<Bar>& bars) {
for (const Bar& b : bars) prefix.push_back(prefix.back() + b.close);
}
// Sum of the closes of bars t-len+1 .. t (compare sums, not divided averages,
// so that the comparison stays exact in integers).
Price sum(int t, int len) const { return prefix[t + 1] - prefix[t + 1 - len]; }
};
// +1 (long) when the fast average is above the slow one, -1 when below,
// 0 while there is not yet enough history.
int signal_at(const Sma& sma, int t, int fast, int slow, int previous) {
if (t + 1 < slow) return 0;
Price f = sma.sum(t, fast) * slow, s = sma.sum(t, slow) * fast; // f/fast vs s/slow
return f > s ? 1 : f < s ? -1 : previous;
}
enum class Fill { NextOpen, SameClose, SameOpenPeek };
struct Costs { Price slippage_ticks = 0; std::int64_t fee_bps = 0; };
struct Execution { int bar; Qty qty; Price price; };
struct Result {
std::vector<Execution> fills;
std::int64_t gross = 0, costs = 0, net = 0, max_drawdown = 0;
std::vector<std::int64_t> pnl; // per-bar change in net equity
};
Result backtest(const std::vector<Bar>& bars, int fast, int slow, Costs c,
Fill mode = Fill::NextOpen) {
const Qty lot = 1;
Sma sma(bars);
Result r;
Qty pos = 0;
std::int64_t cash = 0, peak = 0, last_equity = 0;
int sig = 0, pending = 0;
int t = 0;
auto trade = [&](Qty target, Price price) {
Qty q = target - pos;
if (q == 0) return;
Price paid = price + (q > 0 ? c.slippage_ticks : -c.slippage_ticks);
std::int64_t notional = std::abs(q) * paid;
std::int64_t fee = (notional * c.fee_bps + 9'999) / 10'000; // round up
cash -= q * paid + fee;
r.costs += std::abs(q) * c.slippage_ticks + fee;
pos = target;
r.fills.push_back({t, q, paid});
};
for (t = 0; t < int(bars.size()); ++t) {
const Bar& b = bars[t];
if (mode == Fill::NextOpen) trade(pending * lot, b.open); // decided earlier
// The signal uses this bar's close, so it is known only at the close.
sig = signal_at(sma, t, fast, slow, sig);
if (mode == Fill::SameOpenPeek) trade(sig * lot, b.open); // BUG: the future
if (mode == Fill::SameClose) trade(sig * lot, b.close);
pending = sig;
std::int64_t equity = cash + pos * b.close;
r.pnl.push_back(equity - last_equity);
last_equity = equity;
peak = std::max(peak, equity);
r.max_drawdown = std::max(r.max_drawdown, peak - equity);
}
r.net = last_equity;
r.gross = r.net + r.costs;
return r;
}
// Annualized Sharpe ratio of per-bar PnL, treating a bar as a trading day
// (252 a year, a convention). With a fixed position size this is a ratio
// of PnL mean to PnL spread; the course's lesson 7 defines it on returns.
double sharpe(const std::vector<std::int64_t>& pnl) {
double mean = 0, m2 = 0;
long n = 0;
for (std::int64_t x : pnl) { // Welford
++n;
double d = double(x) - mean;
mean += d / n;
m2 += d * (double(x) - mean);
}
double sd = std::sqrt(m2 / (n - 1));
return sd > 0 ? mean / sd * std::sqrt(252.0) : 0.0;
}
int main() {
auto bars = make_bars(7, 1000);
Result r = backtest(bars, 10, 40, Costs{.slippage_ticks = 1, .fee_bps = 1});
std::println("first fills (bar, qty, price):");
for (const Execution& e : r.fills | std::views::take(4))
std::println(" {:>4} {:>+3} {:>6}", e.bar, e.qty, e.price);
std::println("fills {}", r.fills.size());
std::println("gross PnL {:>5} ticks", r.gross);
std::println("costs {:>5} ticks", r.costs);
std::println("net PnL {:>5} ticks", r.net);
std::println("max drawdown {:>5} ticks", r.max_drawdown);
std::println("Sharpe (ann.) {:>5.2f}", sharpe(r.pnl));
// Accounting check: the per-bar PnL must add up to the final equity.
auto total = std::accumulate(r.pnl.begin(), r.pnl.end(), std::int64_t{0});
std::println("per-bar PnL sums to net: {}", total == r.net ? "yes" : "NO");
}Press Run to compile and run this example.
first fills (bar, qty, price):
40 +1 10034
158 -2 10128
212 +2 10106
290 -2 10195
fills 18
gross PnL 546 ticks
costs 88 ticks
net PnL 458 ticks
max drawdown 103 ticks
Sharpe (ann.) 1.10
per-bar PnL sums to net: yesRead the fills: the first trade buys 1 lot at bar 40, the first bar after 40 closes exist; later trades are 2 lots because they reverse from short to long or back. Each fill price includes the 1-tick slippage. Costs took 88 ticks from a gross PnL of 546.
And the result looks good: a positive net PnL and an annualized Sharpe ratio (mean PnL per bar divided by its standard deviation, times sqrt(252), treating each bar as a trading day; see Returns, Volatility and Risk Metrics) of 1.10. On a series that is pure noise by construction. To be open about it: seed 7 was picked from a handful of seeds because it makes this point, which is itself a small instance of the selection bias this lesson is about. One back-test on one series is a sample of size one. With 1,000 daily bars, the standard error of an annualized Sharpe ratio is about sqrt(252 / 1000) = 0.5 (the independent-returns case in Lo, 2002), so 1.10 is about two standard errors from zero: unusual for one series, and easy to find among a few dozen. The rest of the lesson runs every experiment on hundreds of series, and the averages tell a different story.
Look-ahead bias: one bar of the future is enough
Look-ahead bias means the simulated decision uses information that would not have been available when the trade happened. The most common form is a fill timed one step too early: the signal is computed from a bar's close, and the fill is booked at the same bar's open, hours before that close existed.
The next program runs the same crossover rule on 200 different random-walk series with no costs, under three fill timings:
- next bar's open: honest, as above;
- same bar's close: also honest if you could really trade at the close with a signal computed from it; in practice the close is known only once trading has ended, so this is optimistic;
- same bar's open: the look-ahead bug.
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <print>
#include <vector>
using Price = std::int64_t; // integer ticks
using Qty = std::int64_t; // whole lots
struct Lcg {
std::uint64_t s;
std::uint32_t next() {
s = s * 6364136223846793005ULL + 1442695040888963407ULL;
return std::uint32_t(s >> 33);
}
int range(int lo, int hi) { return lo + int(next() % std::uint32_t(hi - lo + 1)); }
};
struct Bar { Price open, high, low, close; };
// A synthetic random walk in ticks: a gap of -5..+5 ticks from the previous
// close to the open, then four more steps of -5..+5 ticks inside the bar.
// The steps have mean zero, so no rule that looks only at the past can
// expect to make money on it. That is the point.
std::vector<Bar> make_bars(std::uint64_t seed, int n, Price start = 10'000) {
Lcg rng{seed};
std::vector<Bar> bars;
Price p = start;
for (int i = 0; i < n; ++i) {
p += rng.range(-5, 5);
Bar b{p, p, p, p};
for (int step = 0; step < 4; ++step) {
p += rng.range(-5, 5);
b.high = std::max(b.high, p);
b.low = std::min(b.low, p);
}
b.close = p;
bars.push_back(b);
}
return bars;
}
// Simple moving averages from prefix sums: O(1) per query after O(n) setup.
struct Sma {
std::vector<Price> prefix{0};
explicit Sma(const std::vector<Bar>& bars) {
for (const Bar& b : bars) prefix.push_back(prefix.back() + b.close);
}
// Sum of the closes of bars t-len+1 .. t (compare sums, not divided averages,
// so that the comparison stays exact in integers).
Price sum(int t, int len) const { return prefix[t + 1] - prefix[t + 1 - len]; }
};
// +1 (long) when the fast average is above the slow one, -1 when below,
// 0 while there is not yet enough history.
int signal_at(const Sma& sma, int t, int fast, int slow, int previous) {
if (t + 1 < slow) return 0;
Price f = sma.sum(t, fast) * slow, s = sma.sum(t, slow) * fast; // f/fast vs s/slow
return f > s ? 1 : f < s ? -1 : previous;
}
enum class Fill { NextOpen, SameClose, SameOpenPeek };
struct Costs { Price slippage_ticks = 0; std::int64_t fee_bps = 0; };
struct Execution { int bar; Qty qty; Price price; };
struct Result {
std::vector<Execution> fills;
std::int64_t gross = 0, costs = 0, net = 0, max_drawdown = 0;
std::vector<std::int64_t> pnl; // per-bar change in net equity
};
Result backtest(const std::vector<Bar>& bars, int fast, int slow, Costs c,
Fill mode = Fill::NextOpen) {
const Qty lot = 1;
Sma sma(bars);
Result r;
Qty pos = 0;
std::int64_t cash = 0, peak = 0, last_equity = 0;
int sig = 0, pending = 0;
int t = 0;
auto trade = [&](Qty target, Price price) {
Qty q = target - pos;
if (q == 0) return;
Price paid = price + (q > 0 ? c.slippage_ticks : -c.slippage_ticks);
std::int64_t notional = std::abs(q) * paid;
std::int64_t fee = (notional * c.fee_bps + 9'999) / 10'000; // round up
cash -= q * paid + fee;
r.costs += std::abs(q) * c.slippage_ticks + fee;
pos = target;
r.fills.push_back({t, q, paid});
};
for (t = 0; t < int(bars.size()); ++t) {
const Bar& b = bars[t];
if (mode == Fill::NextOpen) trade(pending * lot, b.open); // decided earlier
// The signal uses this bar's close, so it is known only at the close.
sig = signal_at(sma, t, fast, slow, sig);
if (mode == Fill::SameOpenPeek) trade(sig * lot, b.open); // BUG: the future
if (mode == Fill::SameClose) trade(sig * lot, b.close);
pending = sig;
std::int64_t equity = cash + pos * b.close;
r.pnl.push_back(equity - last_equity);
last_equity = equity;
peak = std::max(peak, equity);
r.max_drawdown = std::max(r.max_drawdown, peak - equity);
}
r.net = last_equity;
r.gross = r.net + r.costs;
return r;
}
// Annualized Sharpe ratio of per-bar PnL, treating a bar as a trading day
// (252 a year, a convention). With a fixed position size this is a ratio
// of PnL mean to PnL spread; the course's lesson 7 defines it on returns.
double sharpe(const std::vector<std::int64_t>& pnl) {
double mean = 0, m2 = 0;
long n = 0;
for (std::int64_t x : pnl) { // Welford
++n;
double d = double(x) - mean;
mean += d / n;
m2 += d * (double(x) - mean);
}
double sd = std::sqrt(m2 / (n - 1));
return sd > 0 ? mean / sd * std::sqrt(252.0) : 0.0;
}
int main() {
const int series = 200;
struct Mode { Fill fill; const char* name; };
const Mode modes[] = {{Fill::NextOpen, "next bar's open (honest)"},
{Fill::SameClose, "same bar's close"},
{Fill::SameOpenPeek, "same bar's open (peeks)"}};
std::println("{:<26} {:>10} {:>8} {:>10}", "fill at", "mean gross", "std err",
"positive");
for (const Mode& m : modes) {
double sum = 0, sum_sq = 0;
int positive = 0;
for (int s = 0; s < series; ++s) {
auto bars = make_bars(1000 + s, 1000);
double g = double(backtest(bars, 10, 40, Costs{}, m.fill).gross);
sum += g;
sum_sq += g * g;
positive += g > 0;
}
double mean = sum / series;
double se = std::sqrt((sum_sq - series * mean * mean) / (series - 1) / series);
std::println("{:<26} {:>10.1f} {:>8.1f} {:>6}/{}", m.name, mean, se, positive,
series);
}
}Press Run to compile and run this example.
fill at mean gross std err positive
next bar's open (honest) 1.2 14.6 99/200
same bar's close 6.0 14.6 105/200
same bar's open (peeks) 128.2 13.9 151/200The two honest timings average about zero, within one standard error, and about half the series are positive: exactly what a zero-drift random walk should give. The peeking version makes 128 ticks per series on average, about 9 standard errors above zero, and wins on three series out of four. On noise.
The mechanism is simple. When the rule flips from short to long, it does so because this bar's close pushed the fast average up. A fill at this bar's open buys before that rise, and so collects part of the very move that caused the signal. The bias is not a property of moving averages: any rule computed from a bar and filled within that bar has it.
Look-ahead has other disguises, all worth searching for in a real back-tester:
- using a day's high or low to decide a trade that happens during that day;
- normalizing or scaling the whole series (by its full-sample mean or maximum) before testing;
- using data that was published later than its timestamp (revised figures, corporate events stamped with the event date rather than the announcement date).
The structural defense is the one in the code: a loop in which each step can only read what is already in the past, and a fill price chosen from a later event than the decision. An event-driven back-tester, as in Event-Driven Design, gets this for free because decisions and fills are separate events in time order.
Survivorship bias: testing only the winners
Survivorship bias comes from the choice of data rather than the code. Suppose you back-test on "all the assets in the index today". Those are the assets that survived to today. The ones that fell far enough to be removed (delisted, merged, bankrupt) are missing, and so are their losses. The test universe has been filtered by the outcome you are trying to measure.
The program simulates 500 synthetic assets, each a zero-drift random walk starting at 1,000 ticks, and delists any whose close falls below 700. It then compares buy-and-hold returns over all of them with the survivors alone.
#include <algorithm>
#include <cstdint>
#include <print>
#include <vector>
using Price = std::int64_t; // integer ticks
using Qty = std::int64_t; // whole lots
struct Lcg {
std::uint64_t s;
std::uint32_t next() {
s = s * 6364136223846793005ULL + 1442695040888963407ULL;
return std::uint32_t(s >> 33);
}
int range(int lo, int hi) { return lo + int(next() % std::uint32_t(hi - lo + 1)); }
};
struct Bar { Price open, high, low, close; };
// A synthetic random walk in ticks: a gap of -5..+5 ticks from the previous
// close to the open, then four more steps of -5..+5 ticks inside the bar.
// The steps have mean zero, so no rule that looks only at the past can
// expect to make money on it. That is the point.
std::vector<Bar> make_bars(std::uint64_t seed, int n, Price start = 10'000) {
Lcg rng{seed};
std::vector<Bar> bars;
Price p = start;
for (int i = 0; i < n; ++i) {
p += rng.range(-5, 5);
Bar b{p, p, p, p};
for (int step = 0; step < 4; ++step) {
p += rng.range(-5, 5);
b.high = std::max(b.high, p);
b.low = std::min(b.low, p);
}
b.close = p;
bars.push_back(b);
}
return bars;
}
// 500 synthetic assets. An asset whose close falls below 700 ticks is
// "delisted": it stops trading, and a holder is left with that last price.
int main() {
const int assets = 500, n = 1000;
const Price start = 1000, delist_below = 700;
int delisted = 0;
double sum_all = 0, sum_survivors = 0;
for (int a = 0; a < assets; ++a) {
auto bars = make_bars(5000 + a, n, start);
Price last = bars.back().close;
bool dead = false;
for (const Bar& b : bars)
if (b.close < delist_below) { last = b.close; dead = true; break; }
double ret = double(last - start) / start; // buy-and-hold over the period
sum_all += ret;
if (dead) ++delisted;
else sum_survivors += ret;
}
std::println("assets at the start: {}", assets);
std::println("delisted during the period: {}", delisted);
std::println("mean return, all assets: {:+.2f}%", 100 * sum_all / assets);
std::println("mean return, survivors only: {:+.2f}%",
100 * sum_survivors / (assets - delisted));
}Press Run to compile and run this example.
assets at the start: 500
delisted during the period: 78
mean return, all assets: +0.14%
mean return, survivors only: +5.78%Over all 500 assets, the average return is close to zero, as it must be for zero-drift walks. Among the survivors it is almost 6%. No asset is better than any other; the survivors are simply the ones whose random paths stayed high, and removing the losers removes the left tail of the distribution. Real databases need to include delisted assets, with their last traded prices, to avoid this. When a data source does not say whether it is survivorship-free, assume it is not.
Overfitting: the best of many tries looks like skill
The crossover rule has two parameters. Nothing stops you from trying many pairs and keeping the best. That is where most self-deception in back-testing happens, and it needs no bug at all: each individual back-test is correct. The trouble is the selection. If you try many rules on the same data, the best one is good partly because it fits the noise of that particular sample. This is called overfitting, or data snooping when the data has been reused for many decisions.
The program sweeps 93 pairs (fast lengths 2 to 20, slow lengths 10 to 100) on a random walk of 2,000 bars, with no costs. It picks the pair with the best Sharpe ratio on the first 1,000 bars (in-sample) and then looks at the same pair on the last 1,000 (out-of-sample). Then it repeats the experiment on 100 independent series, and asks how the best in-sample Sharpe depends on the number of pairs tried.
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <print>
#include <vector>
using Price = std::int64_t; // integer ticks
using Qty = std::int64_t; // whole lots
struct Lcg {
std::uint64_t s;
std::uint32_t next() {
s = s * 6364136223846793005ULL + 1442695040888963407ULL;
return std::uint32_t(s >> 33);
}
int range(int lo, int hi) { return lo + int(next() % std::uint32_t(hi - lo + 1)); }
};
struct Bar { Price open, high, low, close; };
// A synthetic random walk in ticks: a gap of -5..+5 ticks from the previous
// close to the open, then four more steps of -5..+5 ticks inside the bar.
// The steps have mean zero, so no rule that looks only at the past can
// expect to make money on it. That is the point.
std::vector<Bar> make_bars(std::uint64_t seed, int n, Price start = 10'000) {
Lcg rng{seed};
std::vector<Bar> bars;
Price p = start;
for (int i = 0; i < n; ++i) {
p += rng.range(-5, 5);
Bar b{p, p, p, p};
for (int step = 0; step < 4; ++step) {
p += rng.range(-5, 5);
b.high = std::max(b.high, p);
b.low = std::min(b.low, p);
}
b.close = p;
bars.push_back(b);
}
return bars;
}
// Simple moving averages from prefix sums: O(1) per query after O(n) setup.
struct Sma {
std::vector<Price> prefix{0};
explicit Sma(const std::vector<Bar>& bars) {
for (const Bar& b : bars) prefix.push_back(prefix.back() + b.close);
}
// Sum of the closes of bars t-len+1 .. t (compare sums, not divided averages,
// so that the comparison stays exact in integers).
Price sum(int t, int len) const { return prefix[t + 1] - prefix[t + 1 - len]; }
};
// +1 (long) when the fast average is above the slow one, -1 when below,
// 0 while there is not yet enough history.
int signal_at(const Sma& sma, int t, int fast, int slow, int previous) {
if (t + 1 < slow) return 0;
Price f = sma.sum(t, fast) * slow, s = sma.sum(t, slow) * fast; // f/fast vs s/slow
return f > s ? 1 : f < s ? -1 : previous;
}
enum class Fill { NextOpen, SameClose, SameOpenPeek };
struct Costs { Price slippage_ticks = 0; std::int64_t fee_bps = 0; };
struct Execution { int bar; Qty qty; Price price; };
struct Result {
std::vector<Execution> fills;
std::int64_t gross = 0, costs = 0, net = 0, max_drawdown = 0;
std::vector<std::int64_t> pnl; // per-bar change in net equity
};
Result backtest(const std::vector<Bar>& bars, int fast, int slow, Costs c,
Fill mode = Fill::NextOpen) {
const Qty lot = 1;
Sma sma(bars);
Result r;
Qty pos = 0;
std::int64_t cash = 0, peak = 0, last_equity = 0;
int sig = 0, pending = 0;
int t = 0;
auto trade = [&](Qty target, Price price) {
Qty q = target - pos;
if (q == 0) return;
Price paid = price + (q > 0 ? c.slippage_ticks : -c.slippage_ticks);
std::int64_t notional = std::abs(q) * paid;
std::int64_t fee = (notional * c.fee_bps + 9'999) / 10'000; // round up
cash -= q * paid + fee;
r.costs += std::abs(q) * c.slippage_ticks + fee;
pos = target;
r.fills.push_back({t, q, paid});
};
for (t = 0; t < int(bars.size()); ++t) {
const Bar& b = bars[t];
if (mode == Fill::NextOpen) trade(pending * lot, b.open); // decided earlier
// The signal uses this bar's close, so it is known only at the close.
sig = signal_at(sma, t, fast, slow, sig);
if (mode == Fill::SameOpenPeek) trade(sig * lot, b.open); // BUG: the future
if (mode == Fill::SameClose) trade(sig * lot, b.close);
pending = sig;
std::int64_t equity = cash + pos * b.close;
r.pnl.push_back(equity - last_equity);
last_equity = equity;
peak = std::max(peak, equity);
r.max_drawdown = std::max(r.max_drawdown, peak - equity);
}
r.net = last_equity;
r.gross = r.net + r.costs;
return r;
}
// Annualized Sharpe ratio of per-bar PnL, treating a bar as a trading day
// (252 a year, a convention). With a fixed position size this is a ratio
// of PnL mean to PnL spread; the course's lesson 7 defines it on returns.
double sharpe(const std::vector<std::int64_t>& pnl) {
double mean = 0, m2 = 0;
long n = 0;
for (std::int64_t x : pnl) { // Welford
++n;
double d = double(x) - mean;
mean += d / n;
m2 += d * (double(x) - mean);
}
double sd = std::sqrt(m2 / (n - 1));
return sd > 0 ? mean / sd * std::sqrt(252.0) : 0.0;
}
struct Pair { int fast, slow; };
struct Score { double in_sample, out_of_sample; };
// Every (fast, slow) pair on one series: Sharpe on bars 0..999 and 1000..1999.
// No costs, so that only luck can make a pair look good or bad.
std::vector<Score> sweep(const std::vector<Bar>& bars, const std::vector<Pair>& grid) {
std::vector<Score> scores;
for (Pair p : grid) {
Result r = backtest(bars, p.fast, p.slow, Costs{});
std::vector<std::int64_t> first(r.pnl.begin(), r.pnl.begin() + 1000);
std::vector<std::int64_t> second(r.pnl.begin() + 1000, r.pnl.end());
scores.push_back({sharpe(first), sharpe(second)});
}
return scores;
}
int main() {
std::vector<Pair> grid;
for (int fast = 2; fast <= 20; fast += 2)
for (int slow = 10; slow <= 100; slow += 10)
if (fast < slow) grid.push_back({fast, slow});
Lcg shuffle{3}; // a fixed random order, so "the first k pairs" is a fair sample
for (int i = int(grid.size()) - 1; i > 0; --i)
std::swap(grid[i], grid[shuffle.range(0, i)]);
// One series: tune on the first half, look at the second.
auto scores = sweep(make_bars(5, 2000), grid);
auto best = std::ranges::max_element(scores, {}, &Score::in_sample) - scores.begin();
int positive = 0;
for (const Score& s : scores) positive += s.in_sample > 0;
std::println("pairs tried: {}, positive in-sample Sharpe: {}", grid.size(), positive);
std::println("best pair ({}, {}): in-sample {:.2f}, out-of-sample {:.2f}",
grid[best].fast, grid[best].slow, scores[best].in_sample,
scores[best].out_of_sample);
// 100 series: how the best of the first k pairs does, for growing k.
const int series = 100;
const int ks[] = {1, 3, 10, 30, int(grid.size())};
double best_in[5] = {}, chosen_out[5] = {};
for (int s = 0; s < series; ++s) {
auto sc = sweep(make_bars(100 + s, 2000), grid);
for (int i = 0; i < 5; ++i) {
auto top = std::max_element(sc.begin(), sc.begin() + ks[i],
[](const Score& a, const Score& b) { return a.in_sample < b.in_sample; });
best_in[i] += top->in_sample / series;
chosen_out[i] += top->out_of_sample / series;
}
}
std::println("{:>6} {:>16} {:>26}", "tried", "best in-sample",
"same pair, out-of-sample");
for (int i = 0; i < 5; ++i)
std::println("{:>6} {:>16.2f} {:>26.2f}", ks[i], best_in[i], chosen_out[i]);
}Press Run to compile and run this example.
pairs tried: 93, positive in-sample Sharpe: 86
best pair (16, 30): in-sample 1.19, out-of-sample 0.27
tried best in-sample same pair, out-of-sample
1 -0.06 0.04
3 0.20 0.03
10 0.40 0.01
30 0.57 0.04
93 0.66 0.06On the first series, 86 of 93 pairs have a positive in-sample Sharpe. That looks like robustness ("almost every setting works"), but it is one series that happened to trend in its first half, and all crossover rules ride the same trend. The pairs are not 93 independent experiments. The best pair's out-of-sample Sharpe is lower, and on a single series that number is itself noise.
The table is the real evidence. Trying one pair gives an in-sample Sharpe around zero, as it should. The more pairs you try, the higher the best in-sample Sharpe climbs: 0.40 after 10, 0.66 after all 93. The out-of-sample Sharpe of the chosen pair stays near zero (its standard error over 100 series is about 0.05) no matter how many pairs were tried. All of the in-sample improvement was selection.
The growth has a familiar shape. The maximum of k independent standard normal draws grows roughly like sqrt(2 * ln(k)). Correlated trials grow more slowly, but the direction is the same. Bailey, Borwein, López de Prado and Zhu (2014) show how quickly this goes wrong for back-tests: with enough trials, a high in-sample Sharpe ratio is almost guaranteed even when every rule has none. A reported back-test result is therefore incomplete without the number of trials that preceded it, including the ones that were discarded.