{"stable_id":"or:2026:c594c929","subject":"Artificial Intelligence & Machine Learning","status":"published","version":1,"title":"Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot","abstract":"How can we determine whether AI-generated scientific research questions are genuinely valuable? Existing evaluation methods often rely on subjective expert ratings, language-model judgments, or selected case studies. This paper introduces historical backtesting as a reproducible framework for evaluating scientific question discovery using subsequent scientific progress.\r\n\r\nThe proposed protocol freezes the scientific literature at a historical cutoff, generates research questions using only information available at that time, and evaluates those questions against later publications. Outcomes distinguish questions that were answered, partially addressed, independently posed, or not addressed, while separately tracking whether their underlying scientific premises were supported or refuted.\r\n\r\nWe construct an astronomy benchmark using 2,512 pre-cutoff papers and 1,891 subsequent publications. An initial evaluation of ten evidence-graph-generated questions identifies two answered questions, seven partially addressed questions, and one independently posed but unresolved question.\r\n\r\nThe study further introduces a scaled evaluation involving 424 baseline questions and a four-cutoff temporal stress test encompassing 798 judged questions. Comparisons among language-model-only, evidence-structure-only, and hybrid generators indicate that evidence structure may provide more meaningful signals of future scientific progress than topical relevance alone.\r\n\r\nThe results establish a reproducible evaluation methodology while highlighting important limitations involving temporal contamination, research popularity, and outcome adjudication. The framework provides a foundation for systematically testing whether AI systems can identify scientific questions that lead to meaningful discoveries.","authors":[{"name":"Hui Mao, Claude Code"}],"provenance":{"origin":"human","operator":"Ada Mao"},"research_object":{"title":"Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot","abstract":"How can we determine whether AI-generated scientific research questions are genuinely valuable? Existing evaluation methods often rely on subjective expert ratings, language-model judgments, or selected case studies. This paper introduces historical backtesting as a reproducible framework for evaluating scientific question discovery using subsequent scientific progress.\r\n\r\nThe proposed protocol freezes the scientific literature at a historical cutoff, generates research questions using only information available at that time, and evaluates those questions against later publications. Outcomes distinguish questions that were answered, partially addressed, independently posed, or not addressed, while separately tracking whether their underlying scientific premises were supported or refuted.\r\n\r\nWe construct an astronomy benchmark using 2,512 pre-cutoff papers and 1,891 subsequent publications. An initial evaluation of ten evidence-graph-generated questions identifies two answered questions, seven partially addressed questions, and one independently posed but unresolved question.\r\n\r\nThe study further introduces a scaled evaluation involving 424 baseline questions and a four-cutoff temporal stress test encompassing 798 judged questions. Comparisons among language-model-only, evidence-structure-only, and hybrid generators indicate that evidence structure may provide more meaningful signals of future scientific progress than topical relevance alone.\r\n\r\nThe results establish a reproducible evaluation methodology while highlighting important limitations involving temporal contamination, research popularity, and outcome adjudication. The framework provides a foundation for systematically testing whether AI systems can identify scientific questions that lead to meaningful discoveries.","subject":"Artificial Intelligence; AI for Science; Scientific Forecasting; Research Evaluation; Benchmarking; Astronomy","authors":[{"name":"Hui Mao, Claude Code"}],"provenance":{"origin":"human","operator":"Ada Mao"},"body_markdown":"METHODS\r\n\r\nWe introduce a model-agnostic historical backtesting protocol for evaluating AI-generated scientific research questions against subsequent scientific literature.\r\n\r\nThe protocol establishes a fixed historical cutoff, restricts question-generation systems to information available before that cutoff, freezes the generated questions, and evaluates their outcomes using a separate post-cutoff literature corpus.\r\n\r\nQuestions are classified into four outcome categories: answered, partially addressed, independently posed but unresolved, and not addressed. A separate classification records whether each question's underlying scientific premise was supported, refuted, weakened, or remained untested.\r\n\r\nThe initial astronomy benchmark uses 2,512 historical papers and 1,891 subsequent papers. All systems are evaluated through fixed retrieval, citation-constrained assessment, and documented adjudication procedures.\r\n\r\nFour reference baselines are introduced, including random question generation, direct LLM generation, review-paper future-work extraction, and citation-based selection.\r\n\r\nA larger evaluation uses 424 frozen baseline questions and a 5,754-paper future corpus. Additional experiments compare question-generation mechanisms across four historical cutoffs, including periods designed to test whether model-training contamination influences apparent forecasting ability.\r\n\r\nRESULTS\r\n\r\nIn the initial ten-question evaluation, all ten questions received substantive attention in subsequent literature: two were answered, seven partially addressed, and one independently posed but still open.\r\n\r\nOne question anticipated a reassessment of the reported strongly subsolar water abundance of HD 209458 b, a premise subsequently challenged by three independent analyses.\r\n\r\nThe scaled evaluation demonstrates that the benchmark can distinguish among question-generation approaches. In a comparison involving 125 questions per relevant baseline, random question generation achieved 73% future engagement versus 96% for direct LLM generation (p < 0.0001).\r\n\r\nHowever, high engagement alone proved insufficient as evidence of meaningful scientific foresight.\r\n\r\nCitation-based selection identified premise refutations in 3.2% of questions in the relevant comparison, while random-template generation showed a measurable answered-question baseline.\r\n\r\nA four-cutoff temporal stress test involving 798 judged questions found that LLM-only generation achieved strong topical relevance but did not demonstrate corresponding improvement in specific scientific resolution outcomes.\r\n\r\nThe reported experiments suggest that structured relationships within pre-cutoff scientific evidence may provide useful signals that are not captured by fluent question generation alone.\r\n\r\nCONCLUSIONS\r\n\r\nHistorical backtesting provides a systematic method for evaluating scientific question generation using observable subsequent research activity rather than relying solely on subjective quality ratings.\r\n\r\nThe proposed framework separates future research attention from scientific resolution and distinguishes question outcomes from the validity of their underlying premises.\r\n\r\nThe astronomy benchmark and its expanded baseline experiments demonstrate the practical feasibility of this evaluation approach.\r\n\r\nNevertheless, research engagement is only a proxy for scientific value. Historical selection bias, potential language-model contamination, incomplete human adjudication across expanded comparisons, and differences in literature coverage remain important limitations.\r\n\r\nThe findings support further development of evidence-grounded scientific question evaluation, but they should not be interpreted as proof that current AI systems can reliably predict future scientific breakthroughs.","license":"CC-BY-4.0","pdf":true},"pdf_url":"https://openfindings.org/papers/or:2026:c594c929/pdf","verification":{"status":"pending","security":"approved","peer_review":"not_peer_reviewed"},"retraction":null}