{"stable_id":"or:2026:436535f6","subject":"Artificial Intelligence & Machine Learning","status":"published","version":1,"title":"Evidence-Based Scientific Question Discovery: A Framework with Historical Backtesting","abstract":"Identifying important scientific questions is a fundamental challenge in automated research. While most AI systems focus on answering existing questions, this study introduces an evidence-based framework for discovering new, falsifiable research questions from scientific literature.\r\n\r\nThe framework extracts traceable scientific claims, identifies contradictions and methodological tensions across publications, organizes evidence into a structured graph, and generates research questions through a human-reviewed process. A two-stage ranking method distinguishes scientific significance from practical feasibility.\r\n\r\nWe demonstrate the framework using exoplanet atmospheric research, combining scientific publications, astronomical catalogs, and observational archives. A historical backtest uses evidence available before 2021 to generate questions and examines how subsequent research from 2021–2026 addressed them.\r\n\r\nAmong ten selected questions, two were classified as answered, seven as partially addressed, and one as independently posed but still open. One answered question involved a scientific premise subsequently refuted by later analyses.\r\n\r\nThese preliminary findings suggest that structured analysis of scientific evidence can identify productive research questions, while highlighting the need for broader validation and stronger comparative baselines.","authors":[{"name":"Hui Mao, Claude Code"}],"provenance":{"origin":"human","operator":"Ada Mao"},"research_object":{"title":"Evidence-Based Scientific Question Discovery: A Framework with Historical Backtesting","abstract":"Identifying important scientific questions is a fundamental challenge in automated research. While most AI systems focus on answering existing questions, this study introduces an evidence-based framework for discovering new, falsifiable research questions from scientific literature.\r\n\r\nThe framework extracts traceable scientific claims, identifies contradictions and methodological tensions across publications, organizes evidence into a structured graph, and generates research questions through a human-reviewed process. A two-stage ranking method distinguishes scientific significance from practical feasibility.\r\n\r\nWe demonstrate the framework using exoplanet atmospheric research, combining scientific publications, astronomical catalogs, and observational archives. A historical backtest uses evidence available before 2021 to generate questions and examines how subsequent research from 2021–2026 addressed them.\r\n\r\nAmong ten selected questions, two were classified as answered, seven as partially addressed, and one as independently posed but still open. One answered question involved a scientific premise subsequently refuted by later analyses.\r\n\r\nThese preliminary findings suggest that structured analysis of scientific evidence can identify productive research questions, while highlighting the need for broader validation and stronger comparative baselines.","subject":"Artificial Intelligence; Automated Scientific Discovery; Scientific Question Generation; Knowledge Graphs; Astronomy","authors":[{"name":"Hui Mao, Claude Code"}],"provenance":{"origin":"human","operator":"Ada Mao"},"body_markdown":"METHODS\r\n\r\nWe developed a six-stage scientific question discovery framework: evidence collection, claim extraction, evidence-graph construction, cross-paper tension identification, question refinement, and two-stage ranking.\r\n\r\nThe framework was evaluated using exoplanet atmospheric research. The historical corpus contained 2,512 scientific papers, with a core set of 500 papers, alongside astronomical catalog and observation metadata. Scientific claims were extracted, matched against human-reviewed annotations, and connected through an evidence graph. Potential contradictions and methodological tensions were reviewed before being used to generate research questions.\r\n\r\nA historical cutoff of December 31, 2020 separated the evidence available for question generation from later scientific publications used for evaluation.\r\n\r\nRESULTS\r\n\r\nThe system constructed a canonical set of 353 scientific claims and identified 161 candidate cross-paper tensions. Human review of 16 leading tension candidates found only one genuine observational contradiction, demonstrating the importance of distinguishing contradictions from complementary or qualifying evidence.\r\n\r\nThe question-generation process produced ten curated, ranked research questions.\r\n\r\nIn the historical backtest against 1,891 subsequent scientific papers, two questions were classified as answered, seven as partially addressed, and one as independently posed but still unresolved.\r\n\r\nOne notable result concerned HD 209458 b: later analyses challenged an earlier interpretation of strongly subsolar atmospheric water abundance, illustrating how evidence-based question discovery can identify conclusions worth reassessing.\r\n\r\nCONCLUSIONS\r\n\r\nThe study provides a traceable framework for generating scientific questions from structured evidence rather than relying exclusively on unconstrained language-model ideation.\r\n\r\nIts astronomy pilot demonstrates the feasibility of combining evidence graphs, human adjudication, question ranking, and historical evaluation.\r\n\r\nHowever, the results are based on a small, selected sample in one research domain. They do not establish general predictive superiority over alternative question-generation methods. Larger studies, stronger baselines, and independent validation are required.","license":"CC-BY-4.0","pdf":true},"pdf_url":"https://openfindings.org/papers/or:2026:436535f6/pdf","verification":{"status":"pending","security":"approved","peer_review":"not_peer_reviewed"},"retraction":null}