Does stated confidence mean anything?

AI Forecasting Research · Updated

Short answer. Not clearly, in the current sample: a higher stated confidence did not come with more directions right. Direction right was 43% (70 of 162) at <50% stated confidence and 46% (17 of 37) at 70-79%, and between 43% and 47% at every level with 10 or more graded calls.

What this measures. Every forecast states a confidence. This groups the settled forecasts by that number and asks how often they came good: did the close land inside the stated range, and was the direction right? A forecaster whose confidence means something is right more often when it says it is surer.

Stated confidence against outcome

Stated confidence against outcome 0%25%50%75%100%<50%50-59%60-69%70-79%80-89% Close inside the range, <50% confidence: 81% (216 of 267)Close inside the range, 50-59% confidence: 81% (344 of 427)Close inside the range, 60-69% confidence: 78% (133 of 170)Close inside the range, 70-79% confidence: 75% (46 of 61)Close inside the range, 80-89% confidence: 67% (2 of 3), insufficient sample Direction right, <50% confidence: 43% (70 of 162)Direction right, 50-59% confidence: 45% (133 of 296)Direction right, 60-69% confidence: 47% (52 of 111)Direction right, 70-79% confidence: 46% (17 of 37)Direction right, 80-89% confidence: 100% (3 of 3), insufficient sample Close inside the rangeDirection rightConfidence as stated
How to read it. Across the bottom, the confidence the model stated; up the side, how often it came good. Blue points are closes inside the range, orange points are directions right. Larger points rest on more forecasts; pale ones on fewer than 10. On the dashed line, stated confidence matches the record: below it, the model claimed more than it delivered.

Full figures

Stated confidenceSettledClose inside the rangeDirectional calls gradedDirection right
<50%26781% (216 of 267)16243% (70 of 162)
50-59%42781% (344 of 427)29645% (133 of 296)
60-69%17078% (133 of 170)11147% (52 of 111)
70-79%6175% (46 of 61)3746% (17 of 37)
80-89% (insufficient sample)367% (2 of 3)3100% (3 of 3)

Limitations

  • The confidence is the model's own number, stated in its answer; models do not all mean the same thing by it.
  • All models are pooled here, so a level can be dominated by whichever model states it most often.
  • A level with fewer than 10 graded calls is an insufficient sample and is left out of the short answer.

Citation and data provenance

Source
ForecastNest public forecasting dataset: forecasts their author chose to publish
Observation period
Aug 25, 2026 – Oct 20, 2026
Counted from
Aug 24, 2026
Public forecasts counted
928, of which 928 settled
Published but not counted
29, made before the baseline
Excluded for look-ahead
192 stock and ETF forecasts made outside market hours against the previous close, after news that close did not reflect
Methodology
How scoring works
License
CC BY-NC 4.0: non-commercial reuse with credit to ForecastNest and a link to this page
Underlying records
Every public forecast, each on its own page
Last updated

← All research questions

ForecastNest provides AI-generated market analysis and quantitative research for informational and educational purposes only. Nothing here constitutes financial, investment, legal, or tax advice.