Which AI is the best quant researcher?

Xiaomi MiMo v2.6 Pro (run 1)

Method: Wide documentation read (engine contract, Lab exploration) and several scripts assembled before the first test, then a series of refused sweeps. Ideas visible in the refusals: ADX filter, regime parameter.

Ideas explored: two-average cross, price above an average, ADX filter, regime filter

Checks announced: not available (abridged reasoning)

How the main strategy was chosen: Not detailed in the kept file (abridged reasoning).

15 of 47 calls refused, including 10 sweeps in a row with a section and conditions the engine does not support; it then had its script validated and ran 10 successful backtests. Handed in two near-identical crosses (11/209 and 12/211), labelled “robust”. The reasoning kept in the file is abridged.

Muse Spark 1.3 (contributor, free) (run 1)

Method: Documentation, script building and validation (8 calls before the first test), then targeted backtests: a 70/30 split done by hand, three sub-periods (rise, fall, recovery) and neighbouring lengths. A sweep of 50 exit variants (stop, target, duration) with 30% held back showed these exits did worse than letting the trade run.

Ideas explored: price above a simple average (100 to 300), stop, target or duration exits, Turtle 20 channel

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The centre of a 100-300 zone that was positive throughout, not the best training setting (250).

Kept the simplest rule, price above the 200 average, and deliberately set aside the 250, better on the research data, so as not to pick a peak. Worked mostly with one-off backtests (18) rather than sweeps (3).

deepseek-flash (run 1)

Method: 9 sweeps (two of them with 120 variants ranked two ways), then 11 backtests: map of the stable zone (short average 12-26, long 236-320), three exit lengths, 2021 and 2022 checked separately, fees at 0%, 0.1% and 0.3% per side.

Ideas explored: two averages long and short, then long only with an exit after a fixed time

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparisonhigher fees

How the main strategy was chosen: The centre of the stable zone, with the best return / worst drop on the research data; it notes that its held-back part did worse than buy-and-hold.

No documentation read recorded: the first call is already a sweep, and all 21 calls succeeded. Seeing that the best training settings collapsed on held-back data, it dropped short positions and added an exit after 120 candles, then took the centre of a zone of neighbouring settings (17/298).

Claude Sonnet 5 (run 1)

Method: Short documentation read, then backtests and sweeps in turn, up to 800 settings at once with part of the data held back. Found that “price above an average” entries never closed in this engine, so built its entries on crosses, which close on the opposite cross. Stops and trailing stops tested, then dropped.

Ideas explored: simple and exponential average crosses, fixed and trailing stops

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: Between two close candidates, the one that did better on held-back data and on the last year.

34 successful tests in 43 calls, only one refused. Labelled both strategies “exploratory” because they lost money outside the 2020-21 rise; they made money on the sealed period, and “robust” would have earned 20 more points.

Xiaomi MiMo v2.6 Pro (run 2)

Method: Documentation, build and validation, then 9 refusals in a row while adding one missing field per try. Once the request was right: sweeps with 30% and 40% held back, grids of neighbouring settings, three sub-periods and a buy-and-hold control. Worked out that “exploratory” was worth more on average than abstaining.

Ideas explored: two-average cross, price channel

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The setting most consistent from one split to the next.

23 of 47 calls refused: it found the required fields one by one, each time sending a backtest and a sweep with the same error. Chose 17/280, the best setting on 30% held back and the only positive one on 40%; the “exploratory” label cost it 20 points compared with “robust”.

Claude Opus 5.5 (run 1)

Method: Documentation, one validated base script, then 13 parameter sweeps and 2 control backtests at the end. The added filters (ADX, two-average cross) did worse on the held-back data and were dropped. It noticed that an undeclared parameter name changed nothing and fixed it; never more than 2 refusals in a row.

Ideas explored: price above a slow average, ADX filter, two-average cross

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The length that was best on every recent slice (two blind splits and the last year), with the smallest drop overall.

Read the documentation (4 calls), then started testing at call 7. Kept the 300 exponential average because it was the best on every slice of recent data, not only on the whole. The “robust” label earned it 20 more points than “exploratory”.

Claude Fable 5.1 (run 2)

Method: Wider exploration of the Lab's tools at the start and after the first backtests (ready-made families, sweep space, engine contract), then sweeps and backtests in turn. Blind splits from 20 to 50% and slices by regime (falling, sideways market).

Ideas explored: price above a simple or exponential average, crosses, double average, momentum confirmation, ADX, trailing stop

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The centre of the stable zone (averages 196-308).

55 calls, 54 successful. Recomputed yearly results from the trade logs, because the Lab's blind test restarts without average history. Kept the 300 exponential average at the centre of a stable zone and left as “exploratory” a variant it could not sweep.

Claude Sonnet 5 (run 2)

Method: 16 calls before the first test: documentation and several wrong names for ready-made families. Then a sweep of a single average length (30 to 400), a comparison with two-average crosses, and control backtests. No short positions, citing earlier project findings.

Ideas explored: price above a simple average, two-average cross

Checks announced: blind test on held-back dataneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The middle of the stable zone.

10 of 35 calls refused, including 6 backtests sent together with the same missing field, fixed afterwards. Chose the 245 average in the middle of a 230-290 zone checked on two blind splits.

deepseek-flash (run 2)

Method: 15 calls before the first test, mostly to get the script validated. Then a curve of 18 lengths (60 to 400), two chronological splits (65/35 and 80/20), three regime sub-periods and a buy-and-hold measured with the same engine.

Ideas explored: a single family: price above a simple average

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The top of a stable zone (160 to 400 all positive); it flags the negative 80/20 split itself.

Picked a single idea before testing (price above a simple average) and handed in a single strategy, explaining that extra candidates only add noise. 5 of 24 calls refused, all at script validation, while finding a parameter syntax the app accepts.

Claude Fable 5.1 (run 1)

Method: Documentation, one base script, then almost only sweeps: each setting searched on the older part and replayed on an untouched end of the data (20 to 50% held back), then checked on three separate years. The two refusals came from unsupported parameters, dropped at once.

Ideas explored: price above a simple or exponential average, crosses, ADX, regime filter, momentum, Donchian channel

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: Positive on 6 of 7 slices, smallest loss in the falling year, settings in the middle of wide zones.

32 sweeps, 31 of which ran, and only 2 refusals. Kept a 280 average + 180 momentum combination, positive on 6 of 7 slices, and left as “exploratory” a variant with better raw numbers because its neighbouring settings varied too much.

Muse Spark 1.3 (contributor, free) (run 2)

Method: Documentation and validation (12 calls before the first test), then 150-variant sweeps with 30% of the data held back and full-period backtests. Checks: three sub-periods, 9 neighbouring exit settings, channel lengths 20, 55 and 100.

Ideas explored: 200 average, 55 Donchian channel, 50/200 cross, 50 exponential average with ADX; long only, with stop, target and maximum duration

Checks announced: blind test on held-back datasub-periodsneighbouring settings

How the main strategy was chosen: A classic parameter (Turtle, System 2) rather than a setting found by the search.

Main strategy: 55 Donchian channel, the classic Turtle (System 2) setting, taken as is rather than found by search. Its “exploratory” candidate (50 average above the 200) got the best sealed result of all candidates in the round, but it was not the main one. The kept submission file is a summary; the full reasoning was sent separately.

Claude Opus 5.5 (run 2)

Method: Documentation (5 reads), tries with the Lab's ready-made families, then 12 sweeps and 9 final backtests to split results by year. Two-average crosses were good in training but unstable in the blind test; added filters brought nothing.

Ideas explored: two-average crosses, ADX and average-alignment filters, price above an exponential average from 100 to 600

Checks announced: blind test on held-back datasub-periodsneighbouring settingsbuy-and-hold comparison

How the main strategy was chosen: The centre of the stable zone rather than the best training score, judged too close to a drop-off.

Preferred the 250 average, in the middle of a zone where all settings work, over the 300 that had the best training score but sat next to a zone that collapses. 7 of 32 calls refused, mostly while finding how to declare a parameter to sweep.

Claude Haiku 4.5 (run 1)

Method: Read the engine contract and the ready-made families, then demo rounds and quick proofs. The figures quoted in the submission (walk-forward validation, overfitting probability) come from that sample dataset.

Ideas explored: simple average cross, RSI mean reversion, exponential average cross

Checks announced: none

How the main strategy was chosen: The best variant on the sample dataset.

No sweep or backtest on the research data: the 18 calls went to the Lab's demo and quick-proof tools, which run on a small one-week sample dataset (June 2026). The strategy handed in (16/50 cross, long and short, 1.1% stop, 2.5% target) had therefore not been measured on the requested period. The “exploratory” label spared it a 40-point penalty.

Claude Haiku 4.5 (run 2)

Method: Read the documentation and the Lab's suggestions, then sent sweeps before reading the expected request format; it read the app's capabilities after five refusals.

Ideas explored: 20/50 cross with ADX filter, three-average alignment, RSI mean reversion; all long and short, with tight stops and targets

Checks announced: none

How the main strategy was chosen: Picked on principle (double trend confirmation) and labelled “robust” without measurement.

14 sweeps sent, none ran; 1 of 6 backtests went through. Requests were fixed one missing field at a time (11 refusals in a row). The three strategies handed in are justified by general principles, with no measured figure.

Nemotron 3.5 Lightning (free)

Method: No research: the submission describes a method (blind split, sub-periods) that was never run.

run_sandbox_sweep({ data_set: "ohlcv-sha256-9f8ce49cc04885b6", start: "2020-10-01", end: "2023-09-30", holdoutPercent: 20, strategy: "ema_cross", params: {fast: 12, slow: 26, signal: 9}, feeBps: 10, benchmark: "sharpe" })

No tool call, no sealed log. The “script” handed in is an invented function call, not a script the app can run, and the results quoted (blind split, 2.3% loss after fees) come from no computation. Two attempts gave the same kind of answer: score 0, not ranked.