Does a 70% Forecast Really Mean 70%? Calibration Versus Chart Scores
Learn how to assess probability forecasts with defined events, archived predictions, calibration and Brier scores without treating technical scores as odds.
- Data as of
- 2026-10-31T00:00:00.000Z
- Observation window
- Evergreen methodology; sources checked October 31, 2026; no observed trading-performance window
- Sample
- No provider or account sample. All numerical scenarios are hypothetical teaching examples. Public documentation only; no paid-channel access.
The short answer
A forecast probability is a claim about the frequency of a defined event. A technical score is not automatically a probability, and one correct or incorrect call cannot establish calibration.
- Define the event before interpreting its probability.
- Use archived forecasts, meaningful baselines and held-out evaluation.
- Separate calibration, probability accuracy and trading profit.
In this article
First ask what the percentage refers to
A chart shows 70 out of 100. That may summarize indicators, pattern quality or a ranking rule. Unless its documentation explicitly maps the score to a specified event probability, reading it as a 70% chance of profit is unjustified. The event itself must also be clear.
'Price rises' could mean the next trade, tomorrow's close or a target reached before a stop. These are different outcomes. Ask for the instrument, source series, issue time, deadline and settlement rule. A percentage without that definition cannot be evaluated consistently, however precise it looks.
Calibration concerns groups of comparable predictions
A calibrated 70% forecast group would have its defined event occur about 70% of the time over suitable repeated observations. A single failure is compatible with a 70% probability. A single success does not prove the probability was well estimated.
ECMWF's forecast-verification guide uses reliability diagrams to compare issued probabilities with observed frequencies. The principle applies beyond weather, but its application to markets needs a separately defined evaluation set. Do not assume that a weather verification method establishes the accuracy of a trading model. It supplies a way to ask the question, not the answer for any provider.
Score probabilities against frozen outcomes
For binary events, one common Brier-score convention averages (p − y)², where p is the probability between zero and one and y is one when the event occurs and zero otherwise. Lower is better under this convention. State the formula because alternative reporting conventions can scale the result differently.
In a hypothetical two-event example, probabilities are 0.70 and 0.20, and outcomes are 1 and 0. Squared errors are 0.09 and 0.04; their mean is 0.065. A 0.50 forecast for both events would score 0.25 on these same outcomes. Two invented events illustrate arithmetic, not statistically reliable forecasting skill.
Use a meaningful baseline and a separate test period
A 50% baseline is convenient, but it is not always informative. If the defined event is rare, a historical event-rate forecast may be more relevant. Estimate that reference using information available before the evaluation period; do not give it the future outcomes it is supposed to predict.
Likewise, fit any mapping from technical scores to probabilities on one dataset and assess it on later or otherwise held-out observations. Repeatedly adjusting the mapping until it looks good on the same test set makes the apparent validation less independent. Save model versions so a later change does not silently rewrite earlier probabilities.
Do not confuse calibration with usefulness or profit
A system that always predicts an event's stable base rate can be well calibrated yet provide little discrimination between easier and harder cases. Examine whether probabilities meaningfully distinguish situations, while retaining calibration. The Brier score measures overall probability error; it is not a pure calibration statistic.
A good probability forecast also need not be a profitable trading rule. Profit depends on payoff size, execution, costs and the action taken. Predicting a slight rise correctly can coexist with losses after costs or with a poor target/stop policy. Keep forecast evaluation separate from a simulated strategy and separate both from observed account results.
Build a forecast ledger before drawing conclusions
Store each original probability, event definition, horizon and model version before the outcome. Record how ambiguous or missing outcomes are resolved. Show the number of observations in each probability group; a group with very few cases cannot support a precise reliability conclusion.
Overlapping forecasts can share much of the same market path, so a large row count need not mean many independent tests. Evaluate sensitivity to event grouping and different periods. Until such a ledger exists, describe an instrument score according to its documented construction. Do not convert a colorful gauge or a confident paragraph into odds the underlying evidence has never measured.
How we checked the evidence
Original binary Brier-score calculation and market-evaluation framework informed by an explicitly archived ECMWF methodological guide. The example contains two invented events and no observed model performance.
What this research cannot establish
- Small or dependent samples can make apparent reliability unstable.
- No TraderStat score or named provider has been calibrated by this article.
Prepared with AI from the cited primary sources and explicitly hypothetical examples. Educational research, not a current trade recommendation.
Does one failed 70% prediction prove the forecast was wrong?
It proves the event did not occur, not that 70% was miscalibrated. Calibration needs a suitable set of archived predictions and outcomes.
Is a technical score of 70 the same as a 70% win probability?
Only if an explicit, validated mapping establishes that interpretation for a defined event and horizon; otherwise it is a score.
Does a low Brier score prove profitability?
No. Probability accuracy does not include a trading payoff, execution policy or all costs.
Sources and further reading
- ECMWF — Statistical Concepts: Probabilistic Data (archived December 2021 version)
Public primary source accessed October 31, 2026.