Having Your Code Audited by Another AI: 25 Defects Confirmed out of 27

A trading strategy judge must be able to be caught out. The fastest way to get there is to hand it to someone with no reason to go easy on it. In October 2026 we handed the Lab, our local backtester and judge, to another AI: Codex. Two audits: one using the Lab over MCP, as any AI client would, the other reading the source code.

Here is the method, the findings and what was fixed. It is also a field report for anyone who has an AI write code: a second model, with a precise brief, finds things the first one does not see.

The brief

Codex had to look for anything that would let a strategy get a "survives" verdict without earning it, and anything that makes a receipt or a published number wrong. Each finding had to come with a file, a line and a numeric scenario. No general impressions: a defect that can be replayed.

Verification: nothing taken on trust

Codex's findings were treated as input data, not conclusions:

• each was checked against a frozen copy of the code (version 245), never modified;
• whenever possible it was reproduced: real MCP calls to the installed Lab, or Rust tests written to pass as long as the defect is present (4 out of 4);
• statistical findings were recomputed independently;
• no proof was run during verification, so that no sealed test period was consumed.

The results

25 findings confirmed out of 27. The severity we assigned sometimes differs from the one Codex proposed: several findings rated "major" turned out to be minor once measured (a shifted descriptive figure, a bounded edge case), and the two refuted findings were intentional refusals, which Codex itself pointed out. Sorting "defect" from "accepted limit" remains the verifier's job.

The five most serious

1. The deflated Sharpe turned into a bonus. The deflated Sharpe ratio penalizes a result according to the number of trials. With an effective number of trials barely above 1 (two nearly identical variants), the approximation of the benchmark became negative: the penalty became a bonus. Reproduced: 99.9999991% instead of 53.98%, on one of the gates of the "survives" verdict. The benchmark was negative for any N between 1 and 1.28.

2. A proof's Pine export changed period. A strategy judged on 2,000 candles was exported to TradingView on 17,971, without its fees or execution policy, while declaring "same strategy". Blocking for anyone trying to reproduce a verdict.

3. An ATR stop could never exist. In the lots engine, a lot bought before the ATR could be computed had no stop. Reproduced: three lots held down to −35.6% despite stop_loss: 2 * ATR(14).

4. "Stop first" was displayed, not applied. When a candle touches both the stop and the target, the cautious option appeared on the receipt, yet 262 lots still exited at the target. A user choosing caution got more optimistic numbers than they thought.

5. The Monte Carlo forgot open positions. Episodes lost the return of still-open lots (product of 1.4545 instead of 1.6). The probability of loss published in reports was wrong, with no effect on the verdict.

Other findings concerned how an AI uses the Lab: the start_here onboarding path led to a Pine export without going through the judge, with hard-coded settings; three arguments recommended by the documentation were rejected by the published schema; a backtest in the full profile returned 189,179 bytes, enough to fill a model's context.

What was fixed

Version 246 brings 21 fixes from these audits:

• Deflated Sharpe: the benchmark never goes below zero and grows with the number of trials. The same case gives 53.22%.
• Pine export: the export keeps the window, fees, execution policy and the fingerprints of the script and data. It only trades from the start of the test period. Check: 2,000 candles exported, and the Pine replays the same trades as the proof (10 out of 10 and 95 out of 95 depending on the engine).
• ATR stop: no purchase until its stop can be computed; first purchase at index 13 in the reproduced case, every lot carries its stop.
• Stop first: applied on CPU and GPU, with bit-identical results.
• Monte Carlo: episodes built from real exposure; the product is back to 1.6 within 10⁻⁹. If conservation fails, no bootstrap: an error status.
• SPA test threshold: now a budget shared by all judged rules, whose sum stays below 10%; when the bootstrap is too coarse to decide at that threshold, the Lab answers "insufficient resolution" instead of deciding.
• MCP onboarding: start_here now goes through the proof before the export, with no hard-coded settings; 82 call examples validated against each profile's schema, 0 failures; a default backtest weighs about 9,600 bytes.

Each fix was followed by non-regression testing: more than 1,000 application tests and more than 1,200 engine tests, with no failure.

The cost of rigor, measured

Stricter gates let fewer false positives through, and fewer real signals too. Both sides were measured on the fixed version, with a preregistered protocol:

• 0 false "survives" out of 872 pure-noise proofs;
• on a planted-signal scenario, the Lab confirms the signal 2 times out of 100, against 11 before the threshold fix.

This cost is published with the rest. We prefer a judge that lets fewer real strategies through to a judge that invents them.

What this teaches about AI-written code

1. A second model finds what the first one misses. The most serious defects were not crashes: the code ran, tests passed, numbers looked plausible.
2. Demand replayable findings. File, line, numeric scenario. A finding that cannot be reproduced does not get fixed, it gets debated.
3. Verify every finding. Here, 25 out of 27 held. The other 2 were deliberate limits. Without verification, you also "fix" what was not broken.
4. A receipt must say what the engine does. Three of the five serious defects were gaps between what was displayed and what was computed.

Limits of this audit: the GPU kernel of the lots engine was not executed during verification (it was for the fixes), and a full proof showing the Monte Carlo defect end to end was not run, so as not to consume a sealed period. Version 246 is built and under verification; the download page still serves 243.

The first defect, found by ourselves, is told here: how our backtest judge caught itself out. The automated research loop, measured on noise: can an AI find a trading strategy?. And to compare AIs with each other: Claude vs ChatGPT vs Grok for trading.

Want your own AI to audit the Lab, or to have it judge your strategies? The MCP guide explains how to plug in Claude Code, Codex, opencode with DeepSeek or a local model. The Lab is free.

Key takeaways

• 27 distinct findings from Codex, 25 confirmed (20 reproduced, 5 by reading), 2 refuted as deliberate limits.
• The most serious: a deflated Sharpe at 99.99% instead of 54% with very few trials; fixed, 53.2%.
• 21 fixes in version 246, including the Pine export that keeps its period and fees.
• Re-measured: 0 false "survives" out of 872 noise proofs; cost published, 2 out of 100 instead of 11 on a planted signal.
• An incorruptible judge is not a judge without errors: it is a judge that gets audited and fixes in public.

Research on past data, not investment advice. The figures above measure a piece of software's behavior on past and simulated data.