How research is recorded and scored
What happens between asking an AI a question and seeing a number next to its answer.
The flow
You pick a symbol and a horizon
We build the prompt. It is the same structure every time and states exactly what to answer — a price range, a confidence, a direction — so two answers to the same question differ because the models differ, not because the questions did. It carries no market data: your AI does its own research, which is why a model that can search the web does better here.
The model has to find the price itself
We do not tell it what the asset is trading at. Establishing that is part of the task, and it is the clearest signal available here of whether a model actually looked anything up: one that cannot say what something costs today has not researched it. Every answer reports the price it found, and that number is kept.
You paste the answer back
A saved forecast cannot be edited, and once settled it cannot be deleted. The window it covers starts when the prompt was built, which is why a prompt expires after an hour — an answer arriving later would be forecasting a period that has partly happened.
You settle it when the window closes
Two prices: what the asset cost when you asked, and what it closed at. The first is the baseline everything is measured from. Asked now rather than at the start, it is a fact about a moment that has passed and can be read off any chart — nobody has to catch a live quote before a prompt expires.
Everyone who answered is settled together
Every model that answered that same question shares a baseline and a window, so they share an outcome, and asking you for the same numbers five times would only invite five chances to mistype them. The window high and low are optional — the close alone is enough to produce a score — but they change what that score can see. Without them the only moment on record is the close. Why you enter the prices rather than us. Both of them, at settlement. Nothing in this loop depends on us buying market data, which is what keeps the research free to run and the record yours. The trade-off is honest: these are self-reported numbers, so a scorecard is a record you keep for yourself. Research other members share is worth reading, and worth judging on its own merits. The one place this could be gamed, and what stops it. The baseline is typed in after the outcome is known, so in principle a number could be chosen to flatter a score. Two things sit against that. It is a public fact — anyone reading shared research can check what the asset cost that day — and every model on the same question independently reported what it thought the price was, printed beside its forecast. A baseline that disagrees with all of them is visible to any reader. Mostly, though, the scorecard is yours: a number bent to look better fools only the person keeping it.
Getting the price wrong
Whatever a forecast scores, it is discounted by how far the model's own price was from the real one. A range that lands near the outcome from a starting point that was wrong is luck, not analysis, and it should not read like analysis.
Under a percent costs nothing. A delayed quote, an extended-hours print, or the minutes between one model answering and the next being asked are not research failures. Past that it falls away smoothly: being 2% out halves the score, 5% takes about four fifths of it, and a model that thinks a $760 index costs a dollar keeps almost none. There is no cliff edge to sit just underneath.
An answer that reported no price at all is not penalised. An unanswered question is not a wrong answer, and the parser that reads these responses can miss a number a model did give — marking a model down for our own bug would be worse than letting one through.
The score
A settlement knows one thing for certain: where the price ended up. So the score is built on the claim a forecast actually makes — which way — and then adjusted by how well that claim was stated.
Direction right 60 – 100 60 for the call, up to 40 more for how it was framed. Direction wrong 0 – 60 How far the price went the other way, relative to how far the forecast said it would go. Zero is reserved — see below.
What earns the other 40
| Component | Weight | What it measures |
|---|---|---|
| Range hit | 35% | Whether the close landed inside the stated range. Missing by a little costs less than missing by a lot. |
| Commitment | 40% | How narrow the range was, measured against the size of the move it predicted. A band wide enough to cover every outcome hits almost every time, so hitting is not what carries a forecast. |
| Confidence | 25% | Whether the model backed its own answer. A right call made at 30% is a weaker result than the same call at 90%. |
Confidence cuts both ways. It earns points on a correct call and costs them on a wrong one. If it only ever deducted, claiming certainty on everything would be the winning strategy — so the worst outcome a forecast can have is being sure and badly wrong, and it is scored that way.
What earns a zero
One thing: a forecast the price never came near. If the range you gave and the range the price actually traded in never overlap at any point in the window, the forecast said nothing true about where price would be, and it scores zero.
Anything the price did touch scores above zero, even when the direction was wrong and it closed somewhere else. A call of $115–$130 on a day price ran from $110.64 to $123.59 was wrong about the direction, but it was not wrong about the level — price spent time inside that band, and the score says so. Closing inside the range is worth more again.
This is why the window high and low are worth entering when you settle. Without them the only moment anyone knows about is the close, so “did price ever reach your range” collapses to “did it finish there” — and a forecast the market passed straight through can end up scored as though it had never been close.
When a range counts as a direction
Add the two ends of the range together, as moves from the price the forecast started at. A range of −6.5% to +6.4% sums to −0.1%; one of −4.1% to +4.8% sums to +0.7%. That sum is the band's lean, and it is zero for a band sitting squarely on the starting price however wide that band is.
A lean of more than 0.2% counts as a call — up if it is positive, down if negative. Below that, no direction was called. A tenth or two of a percent inside a band several percent wide is where the bounds happened to be rounded, not a view about which way price goes, and reading it as one would credit a model for an opinion it never expressed. Those forecasts are still scored, on whether the range held and how wide it had to be, but in a band that stops below 60 — declining to take a side cannot score as well as taking one and being right.
Note what this does not depend on: the width of the range. A model leaning 0.7% up inside a tight band and one leaning 0.7% up inside a wide one have made the same call, and are judged the same way. The width is paid for separately, in how much of the score a correct call actually earns.
What the score is not
It is not a return, a win rate, or anything you could trade on. It measures how well a forecast was stated and whether it held, over one window, for one model. Comparing models across a single question is what the table on each forecast is for; comparing them over time is what your scorecard is for. Neither is advice.
ForecastNest provides AI-generated market analysis and quantitative research for informational and educational purposes only. Nothing here constitutes financial, investment, legal, or tax advice. Conduct your own research and consult a licensed financial advisor before making investment decisions.