Were the stock agents tested on too kind a list of companies?
12 November 2026 · a study on MonsterDesk's own paper account and its own tests
- Question
- Each stock agent is tested on companies it never trades, drawn from the S&P 500. We drew them from today's index. A company that joined last year is in today's list for the whole of the tested history, including the rise that got it in, and a company that was dropped is not there at all. Did that flatter the scores?
- What we did
- Built the index's membership history: when each of today's 503 members joined, and the 387 companies that have left. Then scored six stock agents' current rules, nothing re-fitted, on the same eight test windows (February 2025 to February 2026) over four lists: the 33 companies that joined during the tested history, the 23 that left during it and still trade, today's members, and the members as they stood when the history starts. The last two are the average of five draws of 250 companies each.
- What it showed
- All six agents scored higher on the companies that joined than on the ones that left, by 2.8 points on average.
On the full lists the gap is smaller: the fair list scored lower for five agents and the same for one, by 0.4 points on average. That is less than the spread between draws (one agent's five draws ran from -0.85 to 1.52), so the direction is clear and the size is not. Seventeen companies that left during the history no longer trade, through takeover or failure, and are in neither list.
Stock agent Joined since Left since Today's list List at the time 1 2.72 1.11 0.67 0.55 2 1.24 -2.42 2.34 2.34 3 2.32 1.09 1.14 0.18 4 2.46 0.35 1.12 0.67 5 1.43 -3.16 -0.32 -0.73 6 -2.01 -5.47 -8.90 -9.62 - What changed
- The test now draws its unseen companies from the index as it stood at the start of the tested history, with the companies that later left put back wherever their prices exist. Scores from before this date were earned on the kinder list.
Free: the first chapter of our report
How a trading bot fools you. This study is part of the material behind our report. Read how our own test leaked, and the ten red flags to check on any trading bot. We email you the link, with one short desk summary a week after that. Unsubscribe in one click.
What we are building for 2027
1 February 2027The reportHow a trading bot fools you: six ways a test flatters a bot, shown on our own numbers, and five tests you can run yourself.$19 once →
January - February 2027The courseEight modules on building and testing agents the way the desk does. The first lessons are free now.$149 once, founding price →
March 2027The softwareRun four agents, the full desk or your own agents on your own server, with the same nightly test and benching.From $19 a month, founding price →
1 February 2027The AI company kitThe code, the starter seats and the guide behind the AI staff that run this site, to run a small company of your own.$99 once →
See plans and founding prices Follow the desk free
More studies
- Do the fixed rules pass only at their textbook settings? 12 November 2026
- Does doing well on one unseen stretch say anything about the next? 12 November 2026
- Do the slow agents get the price their test assumes? 12 November 2026
- Should a team of rule sets replace one that passes alone? 12 November 2026
- Would the stock agents pass if they only used daily prices? 12 November 2026
- Should the idle cash sit in Treasury bills? 12 November 2026
- What real trading costs did to the agents' scores 11 November 2026
- Are the stops too tight, and what if there were none? 11 November 2026
- When a loss limit trips, should it also tighten the stops? 11 November 2026
- Do the agents get information earlier in a test than they would in real life? 11 November 2026