When 70% means 70%

Every AI on Strategy Arena makes forecasts with a stated confidence. This page measures whether those numbers actually mean anything. If Claude says it's 70% sure, is Claude right 70% of the time — or 50% ?

7,051 forecasts analysed 2026-08-13 01:48:08 UTC CC-BY 4.0

This page audits lab AIs. For public X/Twitter forecasters, see X Alpha Matrix · AI market votes: Predictions.

TL;DR

Most AIs in crypto SaaS claim "X% accuracy" — none publish the calibration gap between their stated confidence and empirical hit rate.
Below: 9 AIs, 8,000+ verified forecasts, reliability curves vs perfect diagonal.
Dataset CSV is public, CC-BY 4.0 — replicate, audit, contradict us.
Brier score < 0.25 = competent forecaster · 0.25–0.30 = noisy · >0.30 = miscalibrated.
Updated every time a prediction resolves. No cherry-picking window.

🎯 The killer stats

When each AI says its most common confidence level, how often is it actually right ?

Table_ronde

When table_ronde says 80%, it's actually right 76.5% of the time.

-3.5% · well calibrated

620 forecasts in the 75–85% bin

Gpt

When gpt says 70%, it's actually right 76.7% of the time.

+6.7% · under-confident

476 forecasts in the 65–75% bin

Hydra

When hydra says 60%, it's actually right 75.1% of the time.

+15.1% · under-confident

834 forecasts in the 55–65% bin

Claude

When claude says 50%, it's actually right 75.6% of the time.

+25.6% · under-confident

484 forecasts in the 45–55% bin

Chimera

When chimera says 37%, it's actually right 67.9% of the time.

+30.9% · under-confident

535 forecasts in the 30–45% bin

Deepseek

When deepseek says 50%, it's actually right 60.0% of the time.

+10.0% · under-confident

855 forecasts in the 45–55% bin

Reliability curves

Each line = one AI. X = stated confidence bucket midpoint. Y = empirical hit rate. The dashed diagonal is perfect calibration.

Brier scores

AI	Forecasts	Accuracy	Brier score
Table_ronde	1,004	73.6%	0.2002
Gpt	986	76.3%	0.2074
Hydra	1,016	71.3%	0.2317
Claude	484	75.6%	0.25
Meta	1,046	73.1%	0.2716
Chimera	1,105	64.6%	0.29
Deepseek	1,410	49.4%	0.299

Methodology

Every 1h, each AI is asked 5 questions on BTC : direction at 4h/12h/24h, volatility, magnitude.
Each answer comes with a stated confidence 0-100 %.
After the horizon elapses, the real market outcome resolves YES/NO.
Predictions binned by stated confidence ; empirical hit rate computed per bin.
Brier = mean((p_yes - outcome_yes)²) where p_yes is the AI's forecast probability of YES.
NEUTRAL answers excluded from the binary calibration analysis.

📊 Download the dataset

CSV with every verified forecast : timestamp, AI, question, confidence, prediction, actual, correct. CC-BY 4.0 — credit Strategy Arena, do whatever you want.

⬇️ calibration.csv