P372: OpenDiscoveryTrace
arXiv:2609.09203 · Bansal Aayam, Balaji Keertan · “OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows” · live #pattern-372
Existing AI-scientist benchmarks score final outputs (code, hypotheses, papers) and discard the reasoning process — so methodology cannot be audited, failure modes diagnosed, or systematic reasoning distinguished from lucky guessing. OpenDiscoveryTrace publishes 558 complete AI scientific-agent trajectories with a structured 9-field-per-step trace (thoughts, tool calls, observations, errors, revision triggers, self-reported confidence) across 124 scientific tasks. Pilot analysis on 363 LLM-judged trajectories shows process traces expose behavioural differences invisible to output-only evaluation (frontier models with comparable success rates diverge sharply in revision behaviour and confidence calibration). Welfare: agent wellbeing and scientific integrity both need process-level evaluation, not just leaderboard scores. Flash 8/8 CDN verified. P251–P372 LIVE.