The MT5 strategy tester is very good at producing a number that makes you want to trade. That is its failure mode, not its purpose.
A backtest is a hypothesis test where you chose the hypothesis, the data, the instrument, the timeframe, the session and every parameter — after seeing the data. Left alone, it answers "can I find settings that would have worked", and the answer is always yes. The work is in constructing a test that is allowed to say no.
This module is the rulebook we run against our own results, including the two rules we only added after our own results got past the earlier ones.
Tune here, validate there
The protocol is fixed before any run, and it is deliberately boring:
tune 2025-01-01 -> 2026-09-05
validate 2023-01-01 -> 2024-12-31 (never seen by any tune)Validation runs backward in time, which is the opposite of the usual walk-forward instinct and is done for a practical reason: recent data is the data you will inevitably contaminate. You read it, you notice a bad month, you adjust a filter. Older windows can be kept genuinely untouched in a way that "the last six months" cannot.
The detail that makes this more than an intention is where the flag lives. Every run is filed through an ingest script with its tune window attached, and the server decides:
- **In-sample runs** are never shown in the default list. They appear on the version
detail page with an explicit in-sample badge. `IsOutOfSample` is computed from the tune
window, so this cannot be gamed by the caller.Whether a result counts as out-of-sample is not a field you fill in. It is derived from the windows. That matters more than it sounds, because the person most likely to quietly relabel a run as out-of-sample is the person who just tuned it — and that person is you, six weeks later, with a plausible story about why this time it's different.
The run that tested something else entirely
Before any of the statistics matter, there is a more basic question: did the tester run the thing you think it ran?
Two failures from building our own headless test harness, both of which produced clean, plausible, completely meaningless reports:
- `Expert=` must use BACKSLASHES (`KurosawaEA\Engines\RangeRevertEA`). With `/` MT5
silently runs the stock *Moving Average* sample instead (4905 trades, no Inp* inputs).
- `ExpertParameters=` is resolved in `MQL5\Profiles\Tester\`, NOT `MQL5\Presets\`. A set
that is not found is silently ignored and the engine runs on DEFAULTS (first run
filed a bogus `ea-london-rangerevert-usdjpy-m5` pv 0.1.0 row; deleted)Neither of these errors. Both produce a finished report with a profit factor in it. The first tests a sample EA that ships with MetaTrader; the second tests your engine on whatever defaults happen to be compiled in — which, as module 3 established, are not the values you intended to test. The second one reached the database before it was caught.
The fix is not vigilance, it is refusal. The harness now normalises the path, refuses to file a report whose expert name differs from the one requested, and refuses to file a run with zero trades. Both checks are three lines and both have since caught real mistakes.
The general rule: every automated pipeline needs at least one assertion that the output belongs to the input. A backtest that silently tested the wrong EA is worse than one that crashed, because you will act on it.
Modelling quality: disclosed, never gated
Our broker's tick history begins in 2026. Every "real tick" run over 2025–26 is therefore a blend, and anything older is generated ticks only. The published runs say so — Mixed at 7% or 40% real, EveryTick at 0%.
The obvious move is to refuse to publish low-quality runs. We measured instead:
**0 = pure generated ticks, admissible and labelled `EveryTick`** (changed 2026-09-09:
OANDA has no tick history before 2025, and generated vs real ticks moved a 280-trade
M15 result by 2 trades and 0.00 PF)Two trades and no change in profit factor. On this kind of system — closed-bar entries, stops measured in tens of points, held for minutes to hours — tick granularity is not the variable. So the quality figure is published on every run and gates nothing.
The honest version of this is uncomfortable and worth saying: modelling quality would matter enormously for a scalper working inside the spread. The decision is not "tick quality doesn't matter", it is "we measured it for our system and it moved nothing" — and the measurement is the part that transfers, not the conclusion.
Sample size is trades, not years
Every filed run is binned by trade count, and the bin decides how it may be presented:
diagnostic 1-9 trades not published
low_sample 10-29 trades published, labelled "low sample, not evidence"
indicative 30-99 trades published plainly, win or lose
evidence 100+ trades the only tier eligible for `proven`The reasoning behind the boundary between the middle two is the kind of thing that should appear on more results pages:
At n=30 the standard error on win rate is still around
9 points, so an observed 50% could be anywhere from roughly 32% to 68%. Worth reading,
not worth betting on alone.Note what this replaces. "Three years of backtest" sounds substantial and tells you nothing — three years of a once-a-day strategy is a different quantity of evidence from three years of a scalper. Judge a system by trade count, not calendar time, and any retune has to be validated by trade count out of sample too.
The tier is computed on write from the trade count, so it cannot be argued with, and it is recomputed if the thresholds ever change rather than being frozen into old rows.
Nine rules, and the two we had to add
A preset moves from candidate to proven only if all of them hold. Seven are the ones you would expect: a non-overlapping out-of-sample window of at least six months; at least 100 out-of-sample trades; out-of-sample profit factor of at least 1.2 and at least 60% of in-sample; parameter stability under one step in each direction, with anything pinned at a search bound failing outright; costs modelled at or above the live spread cap; the tested build matching the shipped binary; and drawdown inside a number you set in advance.
The interesting two are the ones that were added after presets got past the first seven.
Rule 8 — every window positive (added 2026-09-09):
8. **Every window positive** (added 2026-09-09): no filed test window may have a profit factor
below 1.0, and the most recent window must be positive. Rules 1–7 only compare
out-of-sample against in-sample, so a preset that *lost* over the most recent 20 months
passed on the strength of 2023–24 alone — GBPUSD M15 long-only: OOS PF 1.39, in-sample
PF 0.80. "60% of in-sample" is trivially met when in-sample is bad. This rule is what
stops a good old window from promoting a currently-broken preset.That is a genuinely subtle hole, and it is worth sitting with because the same shape exists in any degradation test. Rule 3 asks whether out-of-sample held up relative to in-sample. If in-sample is bad, an out-of-sample that merely looks better sails through. The preset was, in plain terms, losing money right now and being promoted on a good 2023.
Its three published windows, which you can pull yourself:
2019-2022 PF 0.78 249 trades -$52.00 out of sample
2023-2024 PF 1.39 132 trades +$32.95 out of sample
2025-2026 PF 0.80 85 trades -$14.48 in sampleIt is still on the site, still a candidate, three years later than the window that flattered it.
Rule 9 — edge above cost (added 2026-09-15) closes the other one:
9. **Edge above cost** (added 2026-09-15): on every filed window the mean profit per trade,
after spread and commission as the tester charges them, must be at least one typical
spread of the instrument at the entry time (or, equivalently, the per-trade t-statistic
must be 2 or better). Rule 8 asks only for PF ≥ 1.0, which a preset can satisfy at
$0.015 a trade — NZDJPY fix 0.1.0 did: PF 1.21 / 1.04 on the two windows, 164
out-of-sample trades, +$2.4 total. A profit factor above one with no money behind it is
noise that happened to land on the right side, and it must not carry the site's
strongest label.Both windows profitable. A hundred and sixty-four out-of-sample trades. Two dollars and forty-four cents. Every published rule was satisfied and the result was worth nothing, because the per-trade edge was smaller than the spread it had to pay.
So the rule now requires mean profit per trade, after costs, of at least one typical spread for the instrument — or a per-trade t-statistic of 2 or better — on every filed window. The NZDJPY fix is still published, still a candidate, and is named in the rule as the reference failure.
The pattern behind both additions is the one to take away. Each rule was added the day a real result exposed the gap, and each names the preset that exposed it. A gate written entirely in advance encodes only the failures you could imagine.
The window you did not run
On 2026-09-09 three presets passed all eight rules and were marked proven. The next day they were run on 2019–2022 — a window never used for any decision — and all three came in below 1.0.
Mild, not catastrophic: profit factors of 0.88 to 0.94, drawdowns under 1.5%, but 2021 and 2022 negative on every pair. A Tokyo-morning candidate that had passed the same two windows failed the third at 0.59.
That produced the distinction module 5's kill switch is built around, and it is now part of the gate:
- *Mechanism edge* — passes every available window, including ones spanning different vol and
rate regimes. Earns `proven`, earns the size-up after 30 live trades.
- *Regime edge* — passes the recent windows, fails an older one without being catastrophic
(no window below PF 0.8, no drawdown beyond tolerance). Stays `candidate`. May run live **at
minimum lot only**, as a forward test, and **must carry a kill rule**: rolling profit factor
over the last 30 closed trades below 0.8 pauses the preset.And the requirement changed as a result: the out-of-sample test is now every available prior window, not one. One validation window told us the strategy worked. Two told us which four years it worked in.
Publishing the denominator
Every rejected tune keeps its page and its reason. Not archived, not quietly dropped — published, with the numbers that killed it.
This is the part that is cheap to do and almost nobody does, and the reasoning is short: everyone publishes winners. A results page with no losing entries is not evidence of a good system, it is evidence of a filter, and you cannot tell from outside how wide that filter was. The reject rate is the single most informative number on the page and it only exists if you publish the failures. Ours is currently 18 rejected out of 48 filed versions.
It also costs nothing you did not already spend. The run happened; the reason was written when the decision was made; publishing it is one field.
What you should have now
A tune window and a validation window fixed before the first run, with the out-of-sample flag derived rather than declared; a harness that refuses to file a report it cannot prove came from the right EA; sample size measured in trades; a written gate your results must clear, with a habit of adding a rule the day something slips past; and the losing runs published next to the winning ones.
And the sentence this module exists for: a backtest cannot tell you a strategy works. It can only fail to kill it — and how much that is worth depends entirely on how serious the attempt was.
Next module: from a passing backtest to a running EA — what the first thirty live trades are for, why live and backtest disagree even when nothing is wrong, and the reporting pipeline that makes the comparison possible at all.