Planning the Next Evidence Acquisition Step: An Offline Framework for Active Scientific Discovery

Abstract

Most work on machine-assisted science focuses on generating scientific questions or executing experiments autonomously. We address the step in between: given an existing scientific question, a set of competing hypotheses, and the evidence currently available, what should we investigate next to maximize expected scientific progress? We present an offline, domain-independent planning framework that represents evidence as a typed, confidence-weighted graph; identifies hypotheses with missing, weak, or non-independent evidence; generates generic candidate evidence-acquisition actions; and ranks them by a configurable utility combining expected information gain, novelty, tractability, and cost. Beliefs over hypotheses are revised with a modular Bayesian updater, closing the plan–observe–update loop without ever executing actions. Because the planner is offline, it can be evaluated honestly against history: a benchmark truncates the evidence graph at a time cutoff, hides later evidence as ground truth, and asks whether the recommended actions would have resolved the question earlier. On a worked case study from exoplanet atmospheric characterization — a reported strongly subsolar water abundance whose premise was later refuted by three independent re-analyses — the planner’s top recommendation is exactly the discriminating re-analysis that historically resolved the question (resolution efficiency 1.0; replaying the refuting evidence removes 1.06 bits of hypothesis entropy). We further ground the utility components in the historical-outcome findings of the companion predictor-discovery study (Paper 3, n=980 questions at five cutoffs): explicitly articulated competing hypotheses predict resolution; the resolution channel runs through quantified, explicitly contradictory, recurrent evidence rather than methodological doubt; and semantic novelty does not predict uptake — findings we convert into an empirically calibrated weight configuration that sharpens the planner’s ranking toward the historically resolving actions. Scaling the benchmark to all 374 of Paper 3’s tension questions — each evidence cluster batch-converted to a graph by transparent rules and planned at its own historical cutoff — turns the metrics into distributions and surfaces a structural result: planner utility runs opposite to measured community attention (pooled permutation p = 0.004), because both are driven, with opposite sign, by the breadth of the existing evidence base. We then report the robustness check that qualifies it: the effect does not survive stratification by breadth and concentrates in exactly the high-specificity stratum where Paper 3 documents its retrieval-bounded label to under-count engagement, so the divergence between resolution-driven and attention-driven allocation is real as a mechanism but cannot yet be claimed as detected neglect. The framework is released as a reference implementation with seven evaluation metrics, full configurability, and no domain assumptions in code.