Journal · 9 November 2026

How we stop our AI agents fooling themselves

Any trading strategy can look brilliant on the past. Give a computer enough knobs to turn and it will find a setting that would have made a fortune - on data it has already seen. The hard part isn't finding strategies that worked. It's refusing to trust them until they prove it on data they've never touched.

That trap has a name: overfitting. A model that has memorised the noise of the past instead of learning something durable. It is the main reason most backtests don't survive contact with real markets. Here are the seven checks every MonsterDesk agent has to pass, in the order it faces them.

1. A held-back exam: score only what it has never seen

Each night an agent breeds hundreds of variations of its strategy. History is cut once, in time order, into three parts. The oldest part is the training ground, and the search sees nothing else. The next three months are for choosing: the finalist is the variation that does best there. The newest nine months are held back for the exam, with a gap before each part so nothing leaks across. Only the finalist sits the exam, and how well it fitted the training data counts for nothing.

As time passes a day can move from the exam into training, never the other way. So no strategy is ever examined on a day it was fitted to. And in the exam an agent trades by exactly the rules it trades live: its stop is tested inside each hour, as a resting order at the broker would be, and everything else is decided when the hour closes.

2. The pool exam: it must work on stocks it will never trade

A stock strategy that only works on the fifty names it trades may have simply learned those fifty names. So stock strategies also sit an exam on a fresh sample of 250 S&P 500 companies they will never trade. For slower strategies there is a third exam on twenty years of daily history. The scores are averaged: a rule that only works in its own back yard, or only in the last two years, fails.

3. Beating the champion is deliberately hard

Each agent has a sitting champion strategy. The finalist replaces it only if it beats the champion's exam score by a clear margin and wins at least half of the exam's windows and holds up over the last thirty days. The margin is not fixed. Challenge a champion often enough and some challenger will beat it by luck alone, so every contest in the last thirty days raises the bar for the next one. A new champion then has five days before it can be challenged again, which gives its live record time to count. Most challengers lose. That's the point: an agent should change its mind only when the evidence is strong.

4. The bench: no score, no trading

If an agent's out-of-sample score falls below the bench line, it stops opening positions the same hour. It keeps managing what it holds and keeps learning, and it comes back only when a new version passes the exams. Above the line it trades at full size - an agent that has passed its exams isn't held back - inside the desk-wide loss limits and kill switches that no agent can override.

5. The wobble test: robust or fragile?

A strategy that only works with its settings tuned to the second decimal place has been fitted to noise. So we nudge every numeric setting of each champion by up to 10% of its range, a dozen times, and re-test the copies out of sample. If most of the nudged copies collapse, the champion is marked fragile. We publish that flag, and it is exactly the kind of agent the live check in step 6 is there to catch.

6. Reality wins: live results can overrule the backtest

Once a champion has twenty closed live trades, its real results are compared with what its exam promised. If it is clearly underdelivering - a live Sharpe ratio well below its tested one, or losing trades outweighing winning ones by enough - the agent is benched for a week, whatever its backtest says, and the champion steps down for the next finalist that shows an edge. The agent's own last forty trades are judged too, whoever its champion was, so a run of short-lived champions cannot hide an agent that keeps losing. A bench is served in full: a new champion does not end it. This has already happened: our global simulator's previous champion was demoted after its live results fell far short of its test score.

7. The leak test: is the data from the future?

The most dangerous mistake isn't in a strategy. It's in the data. If a backtest can see information even a few hours before it really became public, every strategy built on it looks smarter than it is. So when a data source changes, we re-run the champions three ways: as normal, with the new data delayed by a full 24 hours, and with it removed entirely. Real skill survives a delay. A timing leak doesn't.

The cause was small and easy to miss: each day's insider filings were stamped at 4:30pm New York time, but filings keep arriving until 10pm. The backtest could occasionally see a filing a few hours before the market could have. Here's what the test showed:

The fix: insider filings now only count from the next morning's market open, and the existing year of history is re-stamped the same way today, before the next market open. stocks-medium is then re-scored honestly at its next learn. Two of the three agents' improvements were real; one was half real. We'd rather publish that than a bigger number.

What the audit found, 11 November 2026

We asked for an independent, line-by-line audit of the code that trains and examines the agents. It found three things we had not.

1. The exam was mostly not unseen. The old exam used rolling windows: train on one stretch, test on the next, many times over. But one strategy was chosen on all the training stretches pooled together, and each test stretch sat inside a later training stretch. Between 83% and 91% of the "unseen" days had been used to choose the strategy. Only the newest six weeks were truly unseen.
2. The exam and the live desk traded different rules. A trade can close four ways: a stop, a target, a time limit, or the signal fading. In the exam the last two never fired, because of one misplaced branch in the code. Live, they fire all the time. On test data the same strategy made 188 trades in the exam and 1,398 the live way. The exam was grading a different strategy from the one that traded.
3. One signal saw the future. The VIX reading for each day was its closing value, and the exam showed it to that same day's trades. The leak test in check 7 covered the newer data sources, not this one.

All three are fixed, and checks 1 and 3 above describe the exam as it now is. The effect was immediate: scores fell as soon as the exam traded the live rules, and more agents went to the bench. Champions chosen under the old exam are marked as such: each stays only until a finalist, chosen without seeing the held-back period, shows an edge on it.

The public record was never affected, because it is the real paper account and not a backtest. What changes is how much weight an exam score deserves. Before this week, less than we claimed.

What none of this can do

All seven checks reduce the odds of fooling ourselves. None of them makes a strategy profitable, and none of them can guarantee that the future will look like the past. That's why the desk runs on simulated money while it builds a public record, why every trade is published, and why the record - not the backtest - is what you should judge it by.

MonsterDesk is analysis software. Nothing here is investment advice, and past results, simulated or otherwise, do not predict future results.