{"stable_id":"or:2026:f35059ae","subject":"Artificial Intelligence & Machine Learning","status":"published","version":1,"title":"Planning the Next Evidence Acquisition Step: An Offline Framework for Active Scientific Discovery","abstract":"Most work on machine-assisted science focuses on generating scientific questions or executing experiments autonomously. We address the step in between: given an existing scientific question, a set of competing hypotheses, and the evidence currently available, what should we investigate next to maximize expected scientific progress? We present an offline, domain-independent planning framework that represents evidence as a typed, confidence-weighted graph; identifies hypotheses with missing, weak, or non-independent evidence; generates generic candidate evidence-acquisition actions; and ranks them by a configurable utility combining expected information gain, novelty, tractability, and cost. Beliefs over hypotheses are revised with a modular Bayesian updater, closing the plan–observe–update loop without ever executing actions. Because the planner is offline, it can be evaluated honestly against history: a benchmark truncates the evidence graph at a time cutoff, hides later evidence as ground truth, and asks whether the recommended actions would have resolved the question earlier. On a worked case study from exoplanet atmospheric characterization — a reported strongly subsolar water abundance whose premise was later refuted by three independent re-analyses — the planner’s top recommendation is exactly the discriminating re-analysis that historically resolved the question (resolution efficiency 1.0; replaying the refuting evidence removes 1.06 bits of hypothesis entropy). We further ground the utility components in the historical-outcome findings of the companion predictor-discovery study (Paper 3, n=980 questions at five cutoffs): explicitly articulated competing hypotheses predict resolution; the resolution channel runs through quantified, explicitly contradictory, recurrent evidence rather than methodological doubt; and semantic novelty does not predict uptake — findings we convert into an empirically calibrated weight configuration that sharpens the planner’s ranking toward the historically resolving actions. Scaling the benchmark to all 374 of Paper 3’s tension questions — each evidence cluster batch-converted to a graph by transparent rules and planned at its own historical cutoff — turns the metrics into distributions and surfaces a structural result: planner utility runs opposite to measured community attention (pooled permutation p = 0.004), because both are driven, with opposite sign, by the breadth of the existing evidence base. We then report the robustness check that qualifies it: the effect does not survive stratification by breadth and concentrates in exactly the high-specificity stratum where Paper 3 documents its retrieval-bounded label to under-count engagement, so the divergence between resolution-driven and attention-driven allocation is real as a mechanism but cannot yet be claimed as detected neglect. The framework is released as a reference implementation with seven evaluation metrics, full configurability, and no domain assumptions in code.","authors":[{"name":"Hui Mao"}],"provenance":{"origin":"human","operator":"Ada Mao"},"research_object":{"title":"Planning the Next Evidence Acquisition Step: An Offline Framework for Active Scientific Discovery","abstract":"Most work on machine-assisted science focuses on generating scientific questions or executing experiments autonomously. We address the step in between: given an existing scientific question, a set of competing hypotheses, and the evidence currently available, what should we investigate next to maximize expected scientific progress? We present an offline, domain-independent planning framework that represents evidence as a typed, confidence-weighted graph; identifies hypotheses with missing, weak, or non-independent evidence; generates generic candidate evidence-acquisition actions; and ranks them by a configurable utility combining expected information gain, novelty, tractability, and cost. Beliefs over hypotheses are revised with a modular Bayesian updater, closing the plan–observe–update loop without ever executing actions. Because the planner is offline, it can be evaluated honestly against history: a benchmark truncates the evidence graph at a time cutoff, hides later evidence as ground truth, and asks whether the recommended actions would have resolved the question earlier. On a worked case study from exoplanet atmospheric characterization — a reported strongly subsolar water abundance whose premise was later refuted by three independent re-analyses — the planner’s top recommendation is exactly the discriminating re-analysis that historically resolved the question (resolution efficiency 1.0; replaying the refuting evidence removes 1.06 bits of hypothesis entropy). We further ground the utility components in the historical-outcome findings of the companion predictor-discovery study (Paper 3, n=980 questions at five cutoffs): explicitly articulated competing hypotheses predict resolution; the resolution channel runs through quantified, explicitly contradictory, recurrent evidence rather than methodological doubt; and semantic novelty does not predict uptake — findings we convert into an empirically calibrated weight configuration that sharpens the planner’s ranking toward the historically resolving actions. Scaling the benchmark to all 374 of Paper 3’s tension questions — each evidence cluster batch-converted to a graph by transparent rules and planned at its own historical cutoff — turns the metrics into distributions and surfaces a structural result: planner utility runs opposite to measured community attention (pooled permutation p = 0.004), because both are driven, with opposite sign, by the breadth of the existing evidence base. We then report the robustness check that qualifies it: the effect does not survive stratification by breadth and concentrates in exactly the high-specificity stratum where Paper 3 documents its retrieval-bounded label to under-count engagement, so the divergence between resolution-driven and attention-driven allocation is real as a mechanism but cannot yet be claimed as detected neglect. The framework is released as a reference implementation with seven evaluation metrics, full configurability, and no domain assumptions in code.","subject":"Artificial Intelligence","authors":[{"name":"Hui Mao"}],"provenance":{"origin":"human","operator":"Ada Mao"},"body_markdown":"**Methods**\r\n\r\nWe introduce an offline, domain-independent framework for planning the next evidence-acquisition step in scientific research. The system represents scientific claims, observations, datasets, papers, and competing hypotheses in a typed, confidence-weighted evidence graph with provenance tracking. It identifies evidence gaps and ranks candidate research actions using a configurable utility function combining expected information gain, provenance-based novelty, tractability, and cost. A modular Bayesian updater revises hypothesis probabilities when new evidence is supplied.\r\n\r\nWe evaluate the framework through historical simulation, withholding post-cutoff evidence from the planner. Evaluation includes a documented exoplanet atmospheric characterization case study and a scaled benchmark of 374 historical evidence-tension questions imported from a companion study.\r\n\r\n**Results**\r\n\r\nIn the exoplanet case study, the planner ranked the historically decisive re-analysis first, achieving a resolution efficiency of 1.0 and a replayed reduction of 1.06 bits in hypothesis entropy.\r\n\r\nAcross 374 historical questions, planner utility was negatively associated with subsequent measured community attention (pooled permutation p = 0.004). However, this association did not remain statistically significant after stratification by evidence-cluster breadth. The remaining association was concentrated in a subgroup affected by documented limitations in historical outcome retrieval.\r\n\r\nThe reference implementation includes seven evaluation metrics and a configurable planning architecture.\r\n\r\n**Conclusions**\r\n\r\nThe study demonstrates a reproducible framework for selecting potentially informative next research steps using structured evidence and competing hypotheses.\r\n\r\nThe results suggest that evidence-based planning and community attention can prioritize different research questions, but do not establish that the planner identifies genuinely neglected scientific opportunities.\r\n\r\nFurther evaluation with richer evidence graphs, full-text-adjudicated outcomes, and more detailed historical action records is necessary before stronger claims about research allocation can be made.","license":"CC-BY-4.0","pdf":true},"pdf_url":"https://openfindings.org/papers/or:2026:f35059ae/pdf","verification":{"status":"pending","security":"approved","peer_review":"not_peer_reviewed"},"retraction":null}