Can an AI Find a Trading Strategy? What 350 Autoresearch Sessions on Pure Noise Show

Letting an AI search for a trading strategy all night is the "autoresearch" idea popularized by Andrej Karpathy: a model proposes a change, tests it, keeps what improves and starts again. On a coding or network-training problem, it works. On market prices the question is different: when the loop says it is improving, has it found something, or has it fitted itself to chance?

We measured the answer on October 11, 2026, on the Lab's autoresearch loop (version 246, still under verification). Here is what was done, what was measured and what remains open.

The test: make the loop search where there is nothing

The most direct way to check a strategy finder is to give it data where no strategy can win, then count how often it announces a discovery.

• Data: synthetic series of 6,000 4-hour candles with no exploitable edge (pure noise, 1.5% volatility per candle, 10 basis point fees).
• Four noise scenarios, depending on the starting point (EMA trend, volume spike, moving-average crossover) and with or without a GPU sweep.
• The loop: 100 steps of 4 mutations each (periods, thresholds, stops, targets, trend filters, SMA or EMA…), about 300 to 400 counted trials per session on CPU, and 3,500 to 4,700 with the GPU.
• 350 sessions in total, each with its own seed.
• Two planted-signal scenarios (weak and clear) to measure the other side: can the loop still find something when there is something to find?

The protocol was preregistered (sha256 fingerprint published) before the final measurements.

What the loop believes

On pure noise, the loop's validation metric improves in 88 to 98% of sessions. Almost every time, the loop "believes" it has made progress. It always has a better version to show in the morning.

That is expected: a loop that tries hundreds of variants and keeps the best one always ends up with a variant that fits the data it sees. This is the mirage machine that needs a frame.

Without trial counting, the statistic runs away

The deflated Sharpe ratio corrects a result for the number of trials used to find it. We computed what each session's best strategy would get if it were judged as a rule written in advance, without counting trials.

Without counting, the sentence "this Sharpe is not luck" would be displayed in two thirds of sessions or more, on data that contains nothing.

With counting and a sealed period: 0 out of 350

The Lab's loop is bound by three rules:

1. Every trial is counted. Every evaluated script is a trial, so is every GPU variant. A duplicate is not a new trial; an abandoned session still counts for later sessions on the same data.
2. The end of the data is sealed before any computation (30% by default, seal written to disk). The loop only sees the training part and selects on an internal validation taken from it. An instrumented test checks this: 0 sealed candles seen by the engine during the session, 1 opening at the end.
3. One final proof. The best rule, never re-selected, is replayed once on the sealed period, as a block walk-forward, in front of the full judge (PBO, deflated Sharpe at the real trial count, Hansen's SPA test against buy-and-hold, CPCV, at least 30 independent events).

Result on the 350 noise sessions: 0 "survives" verdicts (95% confidence interval: 0 to 1.1%).

One useful detail: even without counting, the final verdict already held almost always (0 to 1%), because the sealed period and the other gates stop the mirages. Counting protects the statistical sentence; the seal protects the verdict. Without a seal, judging on the validation the loop itself optimized, a judge that does not count trials says "survives" in 34 to 40% of clear-signal sessions. With counting: 0.

The defect found along the way

The first series of measurements did not hold. In the GPU lane, one session out of 50 got a false "survives" on noise. Replaying that session showed the cause: a rule estimating the "effective number of trials" brought 5,759 counted trials down to 3.4. The kept variants were nearly all twins along the same path, and the rule mistook that similarity for an absence of search.

The fix was written as an addendum to the preregistration (N = all counted trials, never reduced), then re-measured on seeds never used before (101 to 200). That is where 0 out of 350 comes from. The measurements from before the addendum are published as they were, false positive included.

The cost: a safe loop, not yet an efficient one

A strict judge also lets fewer real strategies through. On the clear planted-signal scenario:

• the loop confirms the signal in 8% of sessions (interval 4.1 to 15%);
• the judge alone, applied directly to the starting script, confirms it in 50% of cases.

Two measured causes. First, hundreds or thousands of counted trials make the deflated Sharpe very demanding. Second, the loop selects on a short validation and drifts toward structures (tight stops, filters) that do not hold on the sealed period. At this stage, the loop finds a real signal less often than the judge alone confirms it. It produces no measured false positive, but it is not yet a good researcher. The next ideas (longer or rolling validation, early stopping) still have to be preregistered.

What published research says

Two recent results point the same way:

• An August 2026 paper on strategy discovery by large language models (arXiv 2608.27734) certifies none of the strategies found by two frontier models once trials are counted; only passive benchmarks pass. It also shows that a deliberately leaky "oracle" can pass the deflated Sharpe and the PBO: statistics do not see information leakage, it has to be excluded by construction.
• Language models have memorized part of market history: they reproduce index levels and dates from their training window, and masking names is not enough because they reconstruct them (Lopez-Lira, Tang and Zhu, arXiv 2504.14765). This contamination falls to nearly zero after the model's knowledge cutoff (Gao, Jiang and Yan, arXiv 2512.23847).

The second point is a limit of our own setup: the seal stops the loop from reading the test-period candles, it does not erase the model's memory. An AI that "finds" a rule on 2017-2024 may have read what happened. On our synthetic series the question does not arise; on real markets, the test that counts is a period after the model's cutoff date, or a future period.

Limits of these measurements

• Synthetic data: they measure the false positives and power of the setup, not the profitability of a real strategy.
• The longest measured session is 100 steps. A real 8-hour night has not been timed end to end.
• The autoresearch loop ships with version 246 of the Lab, built and under verification; the download page still serves 243.

So, can an AI find a strategy?

What these measurements allow us to say: an AI that searches will always find something that looks like it works. The question is not whether it finds something, but whether what it tried is counted and whether a period it never saw is kept aside. Without those two rules, the improvement it reports teaches nothing. With them, on noise, nothing gets through.

To compare AIs under the same rules, see Claude vs ChatGPT vs Grok for trading and which AI to use for trading. To plug in your own (Claude Code, Codex, opencode with DeepSeek, a local model) to this judge: the Lab MCP guide. The Lab is free and computes everything on your PC.

On how the judge itself is checked: how our backtest judge caught itself out and having your code audited by another AI. For the original idea: Karpathy's autoresearch.

Key takeaways

• On pure noise, the loop believes it improves in 88 to 98% of sessions.
• Without trial counting, its deflated Sharpe exceeds 95% in 64 to 86% of sessions; with counting, 0 to 1%.
• Counted trials, a sealed period and a single proof: 0 "survives" out of 350 noise sessions.
• The cost is published: on a clear planted signal, the loop confirms it 8 times out of 100, against 50 for the judge alone.
• The seal does not protect against the model's memory: on real data, the test that counts comes after its cutoff date.

Research on past data, not investment advice. The results above are measurements on synthetic and past data, with the limits stated.