Quantitative backtesting is the closest thing trading has to the scientific method. You form a hypothesis, define it in code so there's no room for hindsight to sneak in, run it against historical data, and see whether the results hold up to statistical scrutiny. Finding a genuine edge this way is hard — most ideas that seem obvious in hindsight don't survive contact with real numbers — and that's exactly the point. A backtesting process that never kills any of your ideas isn't rigorous, it's just confirming what you already believed.
The part worth correcting up front: having a real, statistically valid edge does not mean you stop losing money. It means that over a large enough number of trades, the math works out in your favor. Individual trades, weeks, and even months can still lose — sometimes for stretches long enough to make a real edge feel like it stopped working. Understanding that distinction is what separates someone who can survive a live drawdown from someone who abandons a working strategy at exactly the wrong moment, or blows up an account sizing a real edge too aggressively.
Start with a hypothesis, not a script
The most important part of a hypothesis is the why. An edge that can't be explained by some real feature of market structure — where liquidity sits, how a certain type of participant is forced to behave, what a specific data release changes — is much more likely to be a pattern that existed by chance in the specific data you tested it on. That's the core risk in this entire process, usually called overfitting or curve-fitting: tune enough parameters against enough historical data, and you can make almost any strategy look profitable in the past. It will not necessarily work going forward, because you didn't find an edge — you found noise that happened to fit.
Grounding a hypothesis in real research is the best defense against that. If you have an idea — say, that prices tend to reverse sharply at prior session highs and lows — look for the reasoning behind it before building anything. In this example, the reasoning is usually liquidity: stop-losses and pending orders cluster around obvious highs and lows, and larger players who need to fill big orders often target that liquidity deliberately. Academic finance research (SSRN is a good starting point) exists on most of the structural ideas retail traders rediscover independently, and reading it first will save you from testing an idea that's already been shown not to hold up.
One real, well-documented risk worth knowing by name: alpha decay. A published, publicized edge tends to erode over time as more traders act on it and the market adjusts — this is exactly why you should be skeptical of any specific strategy you find fully laid out on YouTube or in a Discord. If it worked and enough people know about it, it's already being arbitraged away.
Get real data, and know what's missing from it
Backtesting quality is capped by data quality. You'll need OHLCV (open, high, low, close, volume) data, and free sources exist — Yahoo Finance, Stooq, and FRED for macro series are common starting points — while paid providers like Polygon.io, Databento, Tiingo, and Norgate Data offer cleaner, longer, and more granular history for serious work.
The mistake almost everyone makes with free data is not checking for survivorship bias. Many free datasets only include companies that are still trading today, quietly excluding every stock that went bankrupt, got delisted, or was acquired out of the sample. A strategy backtested on a survivorship-biased dataset looks meaningfully better than it actually would have performed live, because it never had the chance to hold a stock that went to zero. If you're testing anything that could plausibly hold a failing company, confirm your data source includes delisted names — or discount your results accordingly.
Turning the hypothesis into code
Once the hypothesis and its parameters are defined — entry rules, exit rules, stop-loss and take-profit logic, position sizing — translating it into code is mechanical, and tools like Claude or ChatGPT are genuinely useful here for writing the implementation quickly. What they won't catch for you is look-ahead bias: code that accidentally uses information that wouldn't have actually been available at the time of the trade, like sizing a position off a day's closing price before that day has closed. This is the single most common silent bug in a homemade backtest, and it inflates results without ever throwing an error.
The other detail beginners skip and shouldn't: transaction costs. Commissions, bid-ask spread, and slippage are real, and a strategy that trades frequently can look profitable gross and lose money net once those costs are modeled in. Build them into the backtest from the start rather than bolting them on after the fact — a strategy that only works before costs was never actually a strategy.
The statistics that actually tell you something
A handful of numbers separate a real signal from a story you're telling yourself:
- P-value. The probability of seeing a result this good (or better) purely by chance if there's actually no real edge. A p-value under 0.05 is the conventional threshold for calling a result statistically significant — but it's a convention, not proof, and it comes with a catch: if you test 100 random parameter combinations, roughly 5 of them will clear that bar by chance alone. The more variations you test, the less any single "significant" result actually means. This is called the multiple comparisons problem, and it's a real reason to limit how many parameter combinations you try before you've committed to a hypothesis.
- T-statistic. A rough companion to the p-value; a t-stat above roughly 2.0 generally lines up with statistical significance for a reasonably sized sample.
- R². How much of the variation in returns your model actually explains, versus how much is unexplained noise.
- Win rate and profit factor. Win rate alone is close to meaningless without knowing average win size versus average loss size — a strategy that wins 30% of the time can still be very profitable if wins are large and losses are cut small. Profit factor (gross profit divided by gross loss) captures that relationship in one number; anything consistently above roughly 1.5 is generally considered strong.
- Maximum drawdown. The largest peak-to-trough decline the strategy experienced, in percentage terms — always specify the starting capital you're measuring against, since a $10,000 drawdown means something very different on a $50,000 account than a $500,000 one.
- Sharpe and Sortino ratios. Return per unit of risk taken, measured against the strategy's own volatility. Sortino is the more useful version for most trading strategies, since it only penalizes downside volatility rather than treating a sharp upside move as equally "risky" as a sharp drawdown.
In-sample, out-of-sample, and walk-forward testing
Split your historical data into two pieces. In-sample data is what you use to build and tune the strategy. Out-of-sample data is real historical data the strategy has never seen during development, held back specifically to test whether the edge generalizes — this is not synthetic or simulated data, it's simply data you didn't look at while building the model. A strategy that performs well in-sample and falls apart out-of-sample is very likely overfit.
Walk-forward testing takes this a step further and is worth the extra effort: instead of one in-sample/out-of-sample split, you re-optimize the strategy's parameters periodically as time moves forward — optimize on a window of past data, test on the next unseen chunk, roll the window forward, and repeat. It's a closer simulation of how you'd actually have to run the strategy live, adjusting periodically without ever peeking at the future.
Monte Carlo simulation is a related but separate technique, and it's worth not conflating the two: rather than testing on new data, it takes your existing trade results and randomly reshuffles their order (or resamples them with replacement) thousands of times to show the realistic range of equity curves your edge could have produced by chance alone. It's especially useful for answering an uncomfortable but important question — what's the probability that, purely from normal variance, this strategy could still produce a large enough losing streak to wipe out the account?
Test across as many distinct market regimes as your data allows — a strong bull run, a sharp bear market, a low-volatility grind, a volatility spike like 2020 or 2022. A strategy that only works in one regime isn't broken, but you need to know that going in, not discover it live.
Forward testing, and what a good result actually looks like
Once a strategy clears in-sample, out-of-sample, and walk-forward testing with a real, explainable edge, forward testing — running it live on paper, or with a small real position — is the last check before committing real capital. If you have Claude or another LLM convert your logic into Pine Script, TradingView's native language, you can run and monitor that forward test directly on the platform.
Be honest about sample size here. A month of forward-test results showing a 10% return on a hypothetical $100,000 account, with a profit factor of 2.2, is an encouraging sign — but it's a small handful of trades, and small samples are exactly where randomness looks the most like skill. Treat an early forward-test result as a reason to keep watching, not as final confirmation.
The piece most people skip: position sizing
This is where "having an edge" and "not losing money" actually connect, and it's the part most guides leave out entirely. A real statistical edge, sized too aggressively, can still produce a large enough losing streak to end an account — this is the classic gambler's ruin problem, and it's math, not bad luck. The Kelly criterion gives a formula for the theoretically optimal fraction of capital to risk per trade given your edge and payoff ratio, but full Kelly sizing is aggressive enough that most professional traders run a fraction of it — a quarter or half-Kelly — specifically to survive the variance a real edge still produces.