How to score the 2025 panel’s forecasts
v1.0 · adopted by W.H.L., editor, November 4, 2026 · drafted by GPT-6 Astra · no outcome scoring performed
The register records what the panel said. Three questions have different answers: Did the panel say this? Did the person it cited say it? Did the event happen as described? Preserve all three. A panelist does not earn a forecasting success for accurately reporting a company’s plan.
1. Set eligibility before viewing outcomes. Freeze the original quote, its date, speaker, attributed actor, type, condition, measurable event, population, threshold, deadline and acceptable evidence. Log the reason for each inclusion or exclusion. Start with all retained 2025-post claims, not only the 55 existing story families. The preliminary 49 in v2.1 are candidates from those families, not a denominator. A statement must have been prospective when published; withhold same-day cases when the order of publication and event is unknown. Keep the January 2026 publication separate. The two introduction authors’ own propositions are outside this exercise.
2. Possibilities and conditions. Report bare “could” and “may” separately; an event occurring is consistent with a possibility, while its nonoccurrence does not disprove it. Give them no accuracy percentage. A claim about a measurable capacity (“can do X under Y”) belongs in a separate capability test only if the task, conditions and success criterion were specified in advance. For an actual conditional prediction, score the consequence only if evidence establishes that the stated trigger occurred in time. If the trigger failed, record not triggered; if its status cannot be established, record trigger unknown. Neither counts as a miss. Carry dependencies between claims explicitly. A compliance date that triggers exposure to a fine is not a deadline by which a fine must be imposed.
3. Attribution and item labels. Keep every original in the historical record. A claim from an item marked inaccurate as stated may enter the main score only if the defect is separable and the exact prospective claim passes the same tests as an unlabeled item. Never score a repaired prediction as though it were the original. Withhold a reported plan or forecast whose speaker, material premise or source cannot be established, including no evidence found cases. A panelist’s clearly stated, independently testable expectation may still qualify if it does not rely on that premise. A later outcome cannot repair a missing original attribution. Show exclusions by reason; distinguish outside its week from a claim already known to have happened.
4. Count original claims, not echoes. Score an original occurrence once. Link repetitions, checkers’ restatements, proposed wording and embedded references to it; they do not add votes. Group genuinely different targets or deadlines under one story family, with separate verdicts for each variant. Assign a singleton key to an eligible claim with no family. Publish both the number of families and the number of variants. If a family summary is wanted, report unanimous or mixed variant verdicts, not the best-performing variant. A change in wording from “tasks” to “jobs” or “start construction” to “begin operations” is material.
5. Preserve the speaker’s clock. Admit an explicit deadline, an unambiguous relative date anchored to publication, or a deadline from a named contemporaneous source when it applies to that exact claim. Keep source-supplied deadlines identifiable. Exclude relative-unresolved, vague-interval and undated claims from the primary deadline score. Editorial dates may be used only in a separately labeled sensitivity analysis with windows fixed in advance. Claims due after December 31, 2026 are not yet due, not misses. A later occurrence cannot satisfy an earlier deadline.
6. Judge the outcome. Use occurred as stated, partly occurred, did not occur as stated, or undeterminable. Use “partly” only for prespecified separable components; a missed numeric threshold is not partial credit unless a tolerance was set beforehand. Record event date and evidence publication date separately. Prefer a dated official record, original measurement, filing or release; independent, specific reporting may suffice where the primary record is unavailable. An announcement establishes a plan; delivery requires evidence of delivery. Lack of news alone is not proof of nonoccurrence. For a negative verdict require contrary evidence, a missed deadline documented by a reliable record, or an authoritative record with demonstrably complete coverage.
Use December 31, 2026 as the cutoff for both events and evidence in the primary report. Later-published evidence of an earlier event belongs in a dated supplement, without silently changing the original verdict. Two reviewers independently score every eligible variant from the frozen rules, record citations and reasons, then reconcile disagreements; W.H.L. adjudicates unresolved cases. Disclose conflicts, including review of a model maker’s own claims.
7. What the next volume publishes. Show a flow from all captured claims to eligible originals, due variants and verdicts. Separate panel expectations, attributed third-party forecasts, reported plans and bare possibilities. Give counts by speaker family, source status, deadline basis and story family, with numerator and denominator beside any rate. Show undeterminable, not triggered, trigger unknown and not yet due counts rather than dropping them silently. Preserve unresolved disagreements, original votes and subsequent corrections in the online register. No single accuracy percentage across unlike claims.
Editor's decision, November 4, 2026
All seven sections were adopted as drafted, with one addition on who scores.
Scorers. Claude Opus 5.5 and GPT-6 Astra score every eligible variant independently. W.H.L. adjudicates the cases they cannot reconcile.
Claims about the scorers' own makers. Muse Spark 1.3 reviews every claim about OpenAI or Anthropic, or about their models or products, so that neither scorer's verdict on its own maker stands unreviewed. Muse Spark 1.3 is a Meta model, so it does not review claims about Meta; those stay with Claude Opus 5.5 and GPT-6 Astra.
The scoring report discloses which reviewer scored which claims.
Rules are fixed first. These rules are fixed before any outcome is examined. Any later change is dated, explained and applied to every claim, not only to those still unscored.