Evidence-Based Scientific Question Discovery: A Framework with Historical Backtesting

Research object

METHODS

We developed a six-stage scientific question discovery framework: evidence collection, claim extraction, evidence-graph construction, cross-paper tension identification, question refinement, and two-stage ranking.

The framework was evaluated using exoplanet atmospheric research. The historical corpus contained 2,512 scientific papers, with a core set of 500 papers, alongside astronomical catalog and observation metadata. Scientific claims were extracted, matched against human-reviewed annotations, and connected through an evidence graph. Potential contradictions and methodological tensions were reviewed before being used to generate research questions.

A historical cutoff of December 31, 2020 separated the evidence available for question generation from later scientific publications used for evaluation.

RESULTS

The system constructed a canonical set of 353 scientific claims and identified 161 candidate cross-paper tensions. Human review of 16 leading tension candidates found only one genuine observational contradiction, demonstrating the importance of distinguishing contradictions from complementary or qualifying evidence.

The question-generation process produced ten curated, ranked research questions.

In the historical backtest against 1,891 subsequent scientific papers, two questions were classified as answered, seven as partially addressed, and one as independently posed but still unresolved.

One notable result concerned HD 209458 b: later analyses challenged an earlier interpretation of strongly subsolar atmospheric water abundance, illustrating how evidence-based question discovery can identify conclusions worth reassessing.

CONCLUSIONS

The study provides a traceable framework for generating scientific questions from structured evidence rather than relying exclusively on unconstrained language-model ideation.

Its astronomy pilot demonstrates the feasibility of combining evidence graphs, human adjudication, question ranking, and historical evaluation.

However, the results are based on a small, selected sample in one research domain. They do not establish general predictive superiority over alternative question-generation methods. Larger studies, stronger baselines, and independent validation are required.