Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

Research object

METHODS

We introduce a model-agnostic historical backtesting protocol for evaluating AI-generated scientific research questions against subsequent scientific literature.

The protocol establishes a fixed historical cutoff, restricts question-generation systems to information available before that cutoff, freezes the generated questions, and evaluates their outcomes using a separate post-cutoff literature corpus.

Questions are classified into four outcome categories: answered, partially addressed, independently posed but unresolved, and not addressed. A separate classification records whether each question's underlying scientific premise was supported, refuted, weakened, or remained untested.

The initial astronomy benchmark uses 2,512 historical papers and 1,891 subsequent papers. All systems are evaluated through fixed retrieval, citation-constrained assessment, and documented adjudication procedures.

Four reference baselines are introduced, including random question generation, direct LLM generation, review-paper future-work extraction, and citation-based selection.

A larger evaluation uses 424 frozen baseline questions and a 5,754-paper future corpus. Additional experiments compare question-generation mechanisms across four historical cutoffs, including periods designed to test whether model-training contamination influences apparent forecasting ability.

RESULTS

In the initial ten-question evaluation, all ten questions received substantive attention in subsequent literature: two were answered, seven partially addressed, and one independently posed but still open.

One question anticipated a reassessment of the reported strongly subsolar water abundance of HD 209458 b, a premise subsequently challenged by three independent analyses.

The scaled evaluation demonstrates that the benchmark can distinguish among question-generation approaches. In a comparison involving 125 questions per relevant baseline, random question generation achieved 73% future engagement versus 96% for direct LLM generation (p < 0.0001).

However, high engagement alone proved insufficient as evidence of meaningful scientific foresight.

Citation-based selection identified premise refutations in 3.2% of questions in the relevant comparison, while random-template generation showed a measurable answered-question baseline.

A four-cutoff temporal stress test involving 798 judged questions found that LLM-only generation achieved strong topical relevance but did not demonstrate corresponding improvement in specific scientific resolution outcomes.

The reported experiments suggest that structured relationships within pre-cutoff scientific evidence may provide useful signals that are not captured by fluent question generation alone.

CONCLUSIONS

Historical backtesting provides a systematic method for evaluating scientific question generation using observable subsequent research activity rather than relying solely on subjective quality ratings.

The proposed framework separates future research attention from scientific resolution and distinguishes question outcomes from the validity of their underlying premises.

The astronomy benchmark and its expanded baseline experiments demonstrate the practical feasibility of this evaluation approach.

Nevertheless, research engagement is only a proxy for scientific value. Historical selection bias, potential language-model contamination, incomplete human adjudication across expanded comparisons, and differences in literature coverage remain important limitations.

The findings support further development of evidence-grounded scientific question evaluation, but they should not be interpreted as proof that current AI systems can reliably predict future scientific breakthroughs.