Does the way a prompt is built make a difference?

AI Forecasting Research · Updated

Short answer. In part. Every prompt with 10 or more settled calls or runs lands between 40% (Stock Selection) and 50% (Analyst Revisions) on its own measure. Member research (45%, direction right) and Stock Selection (40%, picks that beat SPY) did clearly worse, while the other 4 cannot yet be told apart from a coin flip. The prompts ask different questions and are scored on different measures, so this says whether each design beats chance, not which design is best. The models differ more than the prompts do. With the prompt held fixed: in Member research, Grok 58% (64 calls) against ChatGPT — GPT-6 32% (50 calls), a gap of 26 points, with Claude — Opus clearly worse than chance and ChatGPT — GPT-6 clearly worse than chance; in Stock Selection, DeepSeek V3 50% (10 runs) against GPT-6 29% (15 runs), a gap of 21 points, though no model there is clearly apart from chance yet; in Event Gaps, Gemini 1.5 Pro 54% (13 runs) against Claude Sonnet 5.5 38% (10 runs), a gap of 16 points, though no model there is clearly apart from chance yet; in Macro Rotation, Grok 4.5 58% (12 runs) against GPT-6 39% (12 runs), a gap of 19 points, though no model there is clearly apart from chance yet; in Extreme Movers, Claude Sonnet 5.5 59% (18 runs) against GPT-6 41% (16 runs), a gap of 18 points, though no model there is clearly apart from chance yet; in Analyst Revisions, Muse Spark 1.1 52% (12 runs) against DeepSeek 50% (12 runs), a gap of 2 points, though no model there is clearly apart from chance yet. The widest gap is in Member research. Multibagger Selection does not yet have two models with 10 or more settled runs to compare. Within member research only the self-research style has 10 or more graded calls, so how a member prompt is built cannot be compared yet.

How each prompt has done overall

How each prompt has done overall Member researchMember research: 45% — Direction right, 609 calls45% (609 calls)Direction right Stock SelectionStock Selection: 40% — Picks that beat SPY, 158 runs40% (158 runs)Picks that beat SPY Event GapsEvent Gaps: 47% — Picks with positive alpha, 148 runs47% (148 runs)Picks with positive alpha Macro RotationMacro Rotation: 49% — Sides called correctly, 98 runs49% (98 runs)Sides called correctly Extreme MoversExtreme Movers: 47% — Directions called correctly, 188 runs47% (188 runs)Directions called correctly Multibagger SelectionNo hit-rate measure — see the table Analyst RevisionsAnalyst Revisions: 50% — Picks with positive alpha, 75 runs50% (75 runs)Picks with positive alpha 50%
How to read it. One bar per prompt: the share of its calls or picks that came good, on that prompt's own measure, named under the bar. Member research counts directional calls; each mode averages over its settled model runs. The measures differ, so compare a bar with 50% rather than with another bar. Pale bars rest on fewer than 10 calls or runs.

Best and weakest model within each prompt

Best and weakest model within each prompt GrokGrok: 58% — Best in Member research · direction right, 64 calls58% (64 calls)Best in Member research · direction right ChatGPT — GPT-6ChatGPT — GPT-6: 32% — Weakest in Member research · direction right, 50 calls32% (50 calls)Weakest in Member research · direction right DeepSeek V3DeepSeek V3: 50% — Best in Stock Selection · picks that beat SPY, 10 runs50% (10 runs)Best in Stock Selection · picks that beat SPY GPT-6GPT-6: 29% — Weakest in Stock Selection · picks that beat SPY, 15 runs29% (15 runs)Weakest in Stock Selection · picks that beat SPY Gemini 1.5 ProGemini 1.5 Pro: 54% — Best in Event Gaps · picks with positive alpha, 13 runs54% (13 runs)Best in Event Gaps · picks with positive alpha Claude Sonnet 5.5Claude Sonnet 5.5: 38% — Weakest in Event Gaps · picks with positive alpha, 10 runs38% (10 runs)Weakest in Event Gaps · picks with positive alpha Grok 4.5Grok 4.5: 58% — Best in Macro Rotation · sides called correctly, 12 runs58% (12 runs)Best in Macro Rotation · sides called correctly GPT-6GPT-6: 39% — Weakest in Macro Rotation · sides called correctly, 12 runs39% (12 runs)Weakest in Macro Rotation · sides called correctly Claude Sonnet 5.5Claude Sonnet 5.5: 59% — Best in Extreme Movers · directions called correctly, 18 runs59% (18 runs)Best in Extreme Movers · directions called correctly GPT-6GPT-6: 41% — Weakest in Extreme Movers · directions called correctly, 16 runs41% (16 runs)Weakest in Extreme Movers · directions called correctly Muse Spark 1.1Muse Spark 1.1: 52% — Best in Analyst Revisions · picks with positive alpha, 12 runs52% (12 runs)Best in Analyst Revisions · picks with positive alpha DeepSeekDeepSeek: 50% — Weakest in Analyst Revisions · picks with positive alpha, 12 runs50% (12 runs)Weakest in Analyst Revisions · picks with positive alpha 50%
How to read it. For each prompt with at least two models on 10 or more calls or runs, its best and its weakest model on that prompt's own measure. The prompt is the same for both, so the gap between them is the difference the model made.
PromptWhat it asksScored onSampleResult so far
Member researchA price range and direction for one asset over a fixed horizon, researched by the model itself.Direction right; close inside the range609 graded calls from 928 settled forecastsDirection right: 45% (275 of 609); in range: 80% (741 of 928)
Stock SelectionFive US stocks per run, each measured against SPY over the same window.Picks that beat SPY; Average return vs SPY158 settled model runsPicks that beat SPY: 40%; Average return vs SPY: -0.43%
Event GapsFive stocks with a material event per run, each forecast from the moment it is asked to the same time one session later and scored in the direction called, against SPY.Picks with positive alpha; Average alpha vs SPY148 settled model runsPicks with positive alpha: 47%; Average alpha vs SPY: -0.18%
Macro RotationThree long and three short ETFs per run from a fixed 14-ETF universe, scored on the long-short spread.Sides called correctly; Average long-short spread98 settled model runsSides called correctly: 49%; Average long-short spread: -0.11%
Extreme MoversFive filtered daily movers per run, scored on the next-session return in the direction the model called.Directions called correctly; Average directional return188 settled model runsDirections called correctly: 47%; Average directional return: -0.22%
Multibagger SelectionFive small and mid-cap stocks per run with a twelve-month thesis, measured against SPY at 1 week, 1 month, 3, 6 and 12 months.Average return vs SPY107 settled model runs (runs measured)Average return vs SPY: -7.47%
Analyst RevisionsFive stocks with a material event per run, each forecast from the moment it is asked to the same time one session later and scored in the direction called, against SPY.Picks with positive alpha; Average alpha vs SPY75 settled model runsPicks with positive alpha: 50%; Average alpha vs SPY: -0.04%

Within member research, by prompt style

Only one prompt style has 10 or more graded calls so far: Self-research — no data supplied, right on direction 45% (275 of 609). There is nothing yet to compare it with.

What this measures. The same direction and range measures, grouped by how the prompt was built: whether it supplied market data, and whether it imposed a method. Pooled across authors: here the prompt is the subject, and the model is not.

Direction right, by prompt style

Direction right, by prompt style Self-research — no data…45% (275 of 609) 50% — a coin flip
How to read it. Each bar is the share of directional calls that were right, with the calls behind it in brackets. The dashed line is 50%, a coin flip. Pale bars marked "insufficient sample" rest on fewer than 10 graded calls. Listed by how many calls were made, not by rank.

Full figures

Prompt styleForecastsSettledDirectional calls gradedDirection rightClose inside the rangeAverage score
Self-research — no data supplied92892860945% (275 of 609)80% (741 of 928)50.5

The prompt styles

Self-research — no data supplied
Supplies no market data; the model researches the asset itself. Leaves the reasoning to the model.

Limitations

  • The prompts ask different questions and are scored on different measures: a share of directions called right and a share of picks that beat SPY are not the same number, so no prompt is ranked above another here.
  • Prompt styles were chosen by the people running the forecasts, not assigned at random, so a difference between styles can come from what each was used for.
  • A style with fewer than 10 graded calls is an insufficient sample and is not compared.

Citation and data provenance

Source
ForecastNest public forecasting dataset: forecasts their author chose to publish
Observation period
Aug 25, 2026 – Oct 20, 2026
Counted from
Aug 24, 2026
Public forecasts counted
928, of which 928 settled
Published but not counted
29, made before the baseline
Excluded for look-ahead
192 stock and ETF forecasts made outside market hours against the previous close, after news that close did not reflect
Methodology
How scoring works
License
CC BY-NC 4.0: non-commercial reuse with credit to ForecastNest and a link to this page
Underlying records
Every public forecast, each on its own page
Last updated

← All research questions

ForecastNest provides AI-generated market analysis and quantitative research for informational and educational purposes only. Nothing here constitutes financial, investment, legal, or tax advice.