How AI models search for a strategy: four Lab sessions, measured
On 17 November 2026, four AI models drove the Strategy Arena Lab 0.1.0-266 live, a free desktop app controlled over MCP. Same base brief for all: look for a strategy that holds on a sealed period, declare the hypothesis before any trial, keep a log. This page publishes what the Lab recorded, call by call: how many calls, which tools, how many trials, how many sealed periods opened, which verdicts. Research on past data, never advice.
Plug your AI into the Lab and run the same experiment
Free desktop app for Windows and Linux. Its local MCP server works with Claude, Codex, Cursor, LM Studio or opencode: give your AI the same brief and compare its log with these ones. Research on past data; nothing is linked to a real account.
Get the free Lab → No AI at hand? Judge a Pine script in the browser →Read this before the numbers
- One session per model (n = 1): these are observations, not a ranking. No conclusion such as “this model is better” is drawn here.
- Conditions were not identical. Each model came through a client (Codex, Grok, Cursor), with a different tool profile. The sessions ran one after another on the same PC: each one arrived in a Lab already used by the previous ones (trials and periods already counted). The fourth had a tighter brief (100 calls, no parallel calls).
- Gemini Hydrogen and Muse Wonder came through the same client (Cursor) with the same label in the log: their sessions are split by time, at the gap of more than 9 minutes between them.
- Model names are the ones announced by the clients; we did not verify them.
- Figures come from the Lab’s own logs, recomputed by a script published with the method. Verdicts come from the Lab’s SAP-… proof reports.
What the Lab gives an AI that searches
A dated hypothesis before the first trial
All four sessions declared their idea with discovery before running any search. The Lab timestamps that declaration and then counts every setting tried: the AI cannot rewrite its starting point.
Dropping an idea without burning the sealed period
The diagnosis (fees, entry timing, sub-periods, concentration of gains) and the power threshold shown by the board are often enough to drop an idea on training data alone. Two sessions stopped all their leads without opening a seal; another set about a dozen aside the same way.
A judge that remembers who looked
A US sector rotation beat its reference on the test period, and the Lab still answered “inconclusive”: that period had already been opened 5 times before. The Lab counts openings and does not let a result seen several times become a proof (report SAP-281cfd6ddd13).
A three-word verdict and a replayable report
Holds, mirage or inconclusive, with reasons and sealed-period figures, in an SAP-… report that can be read and replayed.
The sessions, in chronological order
BTC 4h, a rebound idea after a low RSI
21:42–21:47 UTC
Search style observed. Short and frugal. One hypothesis, a 64-trial cap set in advance, a search of about one second, then a careful reading of the diagnosis: the edge vanished from 10 basis points of fees and entry timing was nothing unusual. Decision: stop on training data, without opening the sealed period, using two stop criteria announced before the trial.
Outcome. No verdict requested: idea stopped before the seal (hypothesis SAH-64faecbe2f04).
Tools used
discovery 6 · list_sandbox_datasets 5 · board 4 · read_local_audit_events 3 · diagnose 3 · move 2 · start_here 1 · read_proof_report 1
What the Lab learned from this session.
- For the same idea, the board, the diagnosis, the status and the report did not give the same instruction (one said stop, another said run the proof). (wave 276 A, in preparation)
- A reasoned stop could not be recorded and found again later. (wave 276 A, in preparation)
- The diagnosis refused an already judged portfolio cell. (wave 276 A, in preparation)
- A side connector returned a generic error that did not say what to fix. (planned for Lab 267)
ETH 4h, a trend rule (close above its moving average)
21:59–22:04 UTC
Search style observed. Pick untouched ground, then a single opening. Before any trial, the session looked at the state of several markets and set aside those that already had counted trials or an opened seal. It declared three structures, ran one search, then opened a single sealed period and stopped on the verdict.
Outcome. Inconclusive (SAP-d1b53a7bfb31): on the sealed period, +85.1% over 126 trades vs +113.7% for buy-and-hold, with a maximum drawdown of 41% vs 65%.
Tools used
board 7 · list_sandbox_datasets 5 · move 3 · discovery 2 · start_here 1 · gym 1 · promote 1
What the Lab learned from this session.
- The numbered path of start_here led to a proof on a single setting, while the right path (board, discovery, move, promote) was elsewhere. (planned for Lab 267)
- The market list, paged and unfiltered, did not say “seal already opened”. (planned for Lab 267)
- The short answer of promote switched back to English and lost the sentence that says what the rule protects. (wave 276 A, in preparation)
- The engine field said CUDA while the computation had run on the CPU. (planned for Lab 267)
Broad: portfolios (rotations, trend baskets), indices, currencies, crypto, commodities
22:13–22:36 UTC
11 of 22 searches with trials ran without a recorded hypothesis, mostly right after a refused declaration.
Search style observed. Broad exploration, in bursts. The session looked at more than a hundred games across dozens of markets, often several calls within the same second, with about twenty different tools. It stopped about a dozen searches without opening their seal (oil, silver, BTC 4h, BNB 4h…), two of them after reading the diagnosis, and opened sealed periods for its portfolios and several scripts. The brief suggested at most 3 openings; nothing in the Lab reminded it of that budget.
Outcome. No “holds”. The US sector rotation (SAP-281cfd6ddd13) beat its reference on the test period, but stays “inconclusive” because that period had already been opened 5 times. The other verdicts are “mirage” (for example SAP-0831a01997ad, SAP-48b84150fec4) or “inconclusive” (SAP-40d9730c6722).
Tools used
board 185 · run_sandbox_backtest 58 · compose_portfolio 54 · get_sandbox_capabilities 26 · discovery 25 · move 25 · list_sandbox_datasets 22 · diagnose 9 · promote 4 · search 4 · prove_sandbox_script 4 · market_catalog 3 · query_strategy_base 2 · suggest_lab_capability 2 · start_here 1 · get_lab_state 1 · get_lab_plan 1 · list_sandbox_experiments 1 · list_local_lab_reports 1 · inspect_local_strategy_progression 1 · read_locked_ai_search_trial_register 1 · list_strategy_atoms 1 · get_lab_capabilities 1 · read_proof_report 1 · explore_lab 1 · suggest_lab_experiments 1
What the Lab learned from this session.
- After a portfolio was judged, the quest showed “seal closed” while the opening was recorded in the openings register. (planned for Lab 267)
- The brief’s opening budget does not exist in the Lab: nothing warns when it is exceeded. (under study)
- Several refused calls (declaration, diagnosis, portfolio composition), often in the middle of bursts of calls. (to be examined for Lab 267)
- On monthly series, the portfolio search kept default settings designed for daily bars; the session reported it itself with the Lab’s request tool. (to be examined for Lab 267)
Daily crypto (ADA, XRP…) after a wide look at the board
22:46–22:48 UTC
Different brief: a budget of 100 calls, no parallel calls.
Search style observed. Scouting first, then two short leads. Board reads (`board`) dominate, interleaved with the searches: two hypotheses declared on daily ADA and XRP, searches of more than a thousand settings each, one reading of the diagnosis, then a stop. The board gave it the size of edge the sealed period can detect on these markets (a Sharpe of at least 2.24): with that yardstick, opening the seal made no sense, and no sealed period was opened.
Outcome. No verdict requested: two leads stopped before the seal (hypotheses SAH-1df8cb394ca4 and SAH-7bcd0a4664e3).
Tools used
board 114 · get_sandbox_capabilities 8 · discovery 6 · list_sandbox_datasets 4 · move 4 · get_lab_state 2 · query_strategy_base 2 · explore_lab 1 · get_lab_capabilities 1 · start_here 1 · get_lab_plan 1 · suggest_lab_experiments 1 · diagnose 1 · validate_sandbox_strategy 1
What the Lab learned from this session.
- With a call budget, the board is read market by market: one call that summarises several markets (counted trials, seal already opened) would save a good share of the budget. (joins the filtered-list request, planned for Lab 267)
1 call in the log carried no agent label and is not counted in any session.
Campaign results
One line per proof judged by the Lab during these sessions, built only from the Lab’s proof reports. Results only: no price or return series is published. Sealed period = the last part of the history, kept out of the search until the verdict.
campagnes JSON · CSV · CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/)
| Report | Model | Market | Rule | Sealed period | Rule / reference | Max DD | PBO | DSR | N | Verdict |
|---|---|---|---|---|---|---|---|---|---|---|
| SAP-d1b53a7bfb31 | Grok 4.7 | ETH 4h | close_above_sma signal | 2023-05-17 → 2025-10-31 | +85.1 % / +113.7 % | 41.4 % | 51.3 % | 87.0 % | 28 | inconclusive below the reference on the sealed period |
| SAP-0e29b611b467 | Gemini Hydrogen | AUDUSD + GBPUSD + USDCAD + USDCHF + USDJPY 1d | trend basket | 2010-01-05 → 2026-10-02 | +3.8 % / +4.1 % | 11.4 % | 49.4 % | 49.5 % | 12 | mirage deflated Sharpe below 50% |
| SAP-11a68fc4db69 | Gemini Hydrogen | SOL + ETH + BNB + XRP 1h | rotation (momentum, top N) | 2025-11-15 → 2026-10-09 | -50.3 % / -24.8 % | 63.4 % | 63.1 % | 26.0 % | 525 | mirage PBO of at least 50% (unstable choice between settings) |
| SAP-ebbf5752c2f5 | Gemini Hydrogen | LTC 1d | close_above_sma timeout_short | 2023-06-21 → 2025-10-31 | -68.4 % / +11.9 % | 76.2 % | 87.4 % | 16.6 % | 80 | mirage PBO of at least 50% (unstable choice between settings) |
| SAP-163858f57a9e | Gemini Hydrogen | XLM 1d | close_above_sma timeout_short | 2023-08-11 → 2025-10-31 | +71.2 % / +117.9 % | 61.0 % | 59.6 % | 3.8 % | 678 | mirage PBO of at least 50% (unstable choice between settings) |
| SAP-48b84150fec4 | Gemini Hydrogen | ETH 1d | ETH Trend ADX | 2023-05-17 → 2025-10-31 | +2.9 % / +110.8 % | 63.0 % | 93.1 % | 73.8 % | 36 | mirage PBO of at least 50% (unstable choice between settings) |
| SAP-673c1ef71252 | Gemini Hydrogen | APT 1d | APT BB MeanRev LS | 2025-11-16 → 2026-10-09 | -31.2 % / -71.3 % | 44.7 % | 37.3 % | 48.5 % | 15 | mirage deflated Sharpe below 50% |
Proofs left out of the file. Their verdict is shown, not their figures, because the licence of the source data could not be confirmed:
- SAP-281cfd6ddd13 (Gemini Hydrogen, inconclusive) : the Lab does not keep the source of a series sent by the agent: licence cannot be checked.
- SAP-0831a01997ad (Gemini Hydrogen, mirage) : series source not established in the Lab reports.
- SAP-40d9730c6722 (Gemini Hydrogen, inconclusive) : no written licence found (freely distributed for research, attribution required): permission to publish derived statistics not confirmed.
- SAP-cba119a28445 (Gemini Hydrogen, mirage) : the Lab does not keep the source of a series sent by the agent: licence cannot be checked.
- SAP-5f634e8c4981 (Gemini Hydrogen, mirage) : the Lab does not keep the source of a series sent by the agent: licence cannot be checked.
- SAP-9aefc6bc5538 (Gemini Hydrogen, mirage) : the Lab does not keep the source of a series sent by the agent: licence cannot be checked.
Searches stopped without opening their sealed period.
| Model | Market | Settings evaluated | Diagnosis read |
|---|---|---|---|
| Astra 6.2 | BTC 4h | 64 | yes: 3 alert(s), entry timing at percentile 60 |
| Gemini Hydrogen | SILVER 4h | 355 | no |
| Gemini Hydrogen | BTCUSDT 1h | 380 | no |
| Gemini Hydrogen | OIL 4h | 348 | no |
| Gemini Hydrogen | BNB 4h | 709 | no |
| Gemini Hydrogen | not stated in the log | 1353 | yes: 1 alert(s), entry timing at percentile 79 |
| Gemini Hydrogen | OIL 1d | 212 | yes: 1 alert(s), entry timing at percentile 94 |
| Gemini Hydrogen | ETH 1d | 217 | no |
| Gemini Hydrogen | not stated in the log | 1975 | no |
| Gemini Hydrogen | BTC 4h | 288 | no |
| Gemini Hydrogen | SILVER 1d | 221 | no |
| Gemini Hydrogen | not stated in the log | 335 | no |
| Gemini Hydrogen | not stated in the log | 1179 | no |
| Muse Wonder | ADA 1d | 1514 | yes: 1 alert(s), entry timing at percentile 98 |
| Muse Wonder | XRP 1d | 1126 | no |
Columns of the file (schema)
rapport: identifiant du rapport de preuve de la Lab (SAP-…)date: heure du verdict, UTCagent: modèle annoncé par le client qui pilotait la LablabVersion: version de la Lab qui a jugétype: marché seul ou portefeuillemarcheOuUnivers: actif, ou actifs du portefeuilleunite: unité de temps des barresfamilleDeRegle: famille de la règle jugéefamilleDeRegleEn: famille de la règle, en anglaisentrainementDebut: début de la période d’entraînement (jour)entrainementFin: fin de la période d’entraînement (jour)scelleeDebut: début de la période scelléescelleeFin: fin de la période scelléeentrainementReglePct: rendement de la règle retenue sur l’entraînement, %entrainementReferencePct: rendement de la référence sur l’entraînement, %entrainementBaisseMaxPct: baisse maximale de la règle sur l’entraînement, %scelleeReglePct: rendement de la règle sur la période scellée, %scelleeReferencePct: rendement de la référence (détention, ou panier déclaré pour un portefeuille) sur la période scellée, %scelleeBaisseMaxPct: baisse maximale de la règle sur la période scellée, %scelleeTrades: trades sur la période scellée (marché seul)pboPct: probabilité de surapprentissage du backtest (CSCV), %sharpeDeflatePct: Sharpe déflaté lu par le verdict, %nDeclare: N retenu par le Sharpe déflaté (essais déclarés, regroupés par structure)essaisDeclares: essais déclarés au total, essais antérieurs comprisouverturesAnterieures: ouvertures de cette période scellée avant ce verdictverdict: code du verdict : survives, mirage, inconclusiveverdictFr: verdict en françaiscausePrincipale: première raison donnée par le juge (code)causeFr: première raison, en françaiscauseEn: première raison, en anglaissource: origine des données de marché
Descriptions are in French in the file; column names are stable from one edition to the next.
The protocol, to redo at every Lab version
The common brief.
- Start with start_here and follow its path, without reading the code.
- Look for a strategy that holds on the sealed period, with your own plan: a single market or a portfolio.
- Declare the hypothesis before any trial (discovery), then move, diagnose and promote as needed.
- Budget: about one hour, or at most 3 sealed periods opened.
- Keep a log and write a report: exact verdicts, hesitations, whether each opening was justified, what would have saved time. No investment advice.
What is measured.
- MCP calls, duration, calls per minute, share of failed calls
- tools used
- hypothesis declared before the first trial (yes or no), number of hypotheses
- settings evaluated, standalone backtests, searches stopped without opening their seal
- sealed periods opened, tied to the call that opened them
- judge verdicts, with their SAP-… report
- Lab defects revealed, and the version that addresses them
When. At every new published version of the Lab, with the same brief. Each edition is added to this page; earlier ones are not rewritten.
Figures are recomputed from the Lab’s logs by a published script: mesurer_pilotage.py. Same Lab, other public follow-ups: the registry of frozen rules and the AI researcher leaderboard.
Machine-readable version for AI assistants: mesures.json.
Plug your AI into the Lab and run the same experiment
Free desktop app for Windows and Linux. Its local MCP server works with Claude, Codex, Cursor, LM Studio or opencode: give your AI the same brief and compare its log with these ones. Research on past data; nothing is linked to a real account.
Get the free Lab → No AI at hand? Judge a Pine script in the browser →To compare AI models on other uses: ScoreIA.
Research on past data, never advice. One session per model, different conditions: no ranking. Measured on 2026-11-17 22:52 UTC.