Models
| Paper | Models | Agent setup |
|---|---|---|
| Fork-and-Flush | GPT-5.6-terra (high); GPT-5.5 (extra-high) for VLIW and tinyCIFAR. | Copilot CLI; four branches per fork. |
| Forgetting | DeepSeek V4.1 Flash. | Pi; four communicating agents with periodic resets. |
| SLA | GPT-5.5 or Opus 4.8. | Codex or Claude Code; stateless Advisor and Workers. |
| RSI-top | DeepSeek v4 Flash and Gemini 3.7 Flash for kernel search; Gemini 3.7 Flash for MLE-bench. | Frozen models; dashboard and selective history retrieval around OpenEvolve/ShinkaEvolve, plus diverse intervention plans. A ReAct setup is used for MLE-bench. Kernel wrapper code. |
| CCTS | Gemini 2.0 Flash. | Program generation and concept extraction; statistical parent selection. |
| MIRA / MIRA-AC | Qwen3.6-27B for MIRA-AC; GPT-5.5 and GPT-6 Astra in untrained MIRA studies. | Trained decision policy and shared critic; fixed curator and fresh Codex executors. |
| IdeaScientist | Qwen3.6-27B; base variants include Gemma-4-31B and GPT-OSS-120B. | GRPO-trained ideation roles; fixed orchestrator. |
| Evolving in Thought Space | Qwen3-8B/14B guidance; GLM-5.2 executor. GPT-OSS-120B ablations. | Guidance-only LoRA; PUCT archive. |
| DuoDiscover | Opus 5 with Sonnet 5 helper; Gemini 3.8 Flash in additional runs. | Claude Code or Antigravity; four Workers by default. |
Evaluation sample complexity
One example per paper. An evaluation means scoring a candidate with the task evaluator; proposals, internal tests and final grading are counted separately.
| Paper | Selected example | Evaluation count | Where proposals and evaluations differ |
|---|---|---|---|
| Fork-and-Flush | Polyomino, Frontier-CS #0; 25,000 credits/run. | Whole-run total not reported. An agent episode can submit for grading multiple times. | Four branches do not imply four evals. The public task config lists 70 test cases per full suite; those are cases within an eval, not 70 new candidates. The paper’s search wrapper is unavailable for verification. |
| Forgetting | AHC014, four-agent continue-vs-forget ablation. | 30 prefix + 30 continuation proposals per agent: 240 proposals across the team, including the shared prefix. Completed eval total not reported. | The harness scores candidates after an agent returns; the agent is told not to run the judge itself. Empty turns can produce no candidate. The proposal budget alone does not establish how many candidates reached the judge. |
| SLA | VLIW, Codex/GPT-5.5; 1B-token budget. | 3 Workers × 3 local trials = 9 scheduled trials per full Advisor epoch. Whole-run evaluator-call total not reported. | The harness evaluates the returned candidate outside the Worker. Nine trials are not evidence of nine completed evals: failure, retry and caching behavior require the implementation or logs, which are not released in the official repository. |
| RSI-top | Anthropic VLIW kernel; 12-hour search. | Released wrapper: 4 planners × 8 directions = 32 proposals per intervention, before deduplication. No fixed whole-run evaluation total. | The code deduplicates directions before execution. Failures can prevent scoring; native-search evaluations add to intervention evaluations. Thirty-two proposals do not imply thirty-two completed evals. |
| CCTS | Circle packing, n = 26; main Gemini-2.0-Flash comparison. | 25 search iterations; the defined loop schedules one child-fitness evaluation per iteration. | This gives 25 scheduled calls to the program-level evaluator, not an audited invocation total. Seed evaluations, failed-generation retries and optimization inside the generated program are not itemized. The 60 runs are independent replicates. |
| MIRA / MIRA-AC | BNLearn causal discovery; one 14–34-node graph. | Up to 20 iteration-end SHD scores + a final graph-scoring stage, derived from the protocol. Separately, at most n interventions on an n-node task. | The executor cannot query graph accuracy during a work order. The harness reports SHD after each completed iteration. BIC diagnostics and interventions are separate queries, so the 20-decision cap does not cap every kind of evaluation at 20. |
| IdeaScientist | One proposal-generation run, then the standard evaluation script. | In-loop review count is adaptive. For a valid final report, the default scorer makes 5 logical rubric calls, before retries and optional metrics. | The 100-turn orchestrator cap is not a review cap: a turn can spawn multiple sub-agents. Reviews use literature and judge reasoning; the proposed scientific experiment is not executed. The five final calls are outside the search loop. |
| Evolving in Thought Space | Polyomino, Frontier-CS #0. | 30 × 8 parents × 16 rollouts = 3,840 attempts. | Parse/compile failures consume attempts without completing evaluation. Seed and final re-evaluations are separate. |
| DuoDiscover | vLLM serving on 8×B200, 20 hours, initially four agents. | About 90 candidate edits; 7 full true evaluations, including the final combined edit. Claude Code uses 12 full evals. | Cheap yardsticks screen candidates first; their invocation total is not reported for this run. One full eval bundles correctness checks and a serving-load sweep over four metrics (~85 min). Those metrics are not four separate candidate evals. |
Code checks and counting details
IdeaScientist is directly traceable in code. The runner loops over tool calls within each orchestrator turn and executes the requested sub-agents. A reviewer invocation can itself take several LLM turns. For a valid report and available comparison records, evaluate_run requests five rubrics: proposal quality, relevance, novelty against all work, novelty before the cutoff, and reference-grounded similarity. Each is one logical judge call. The JSON parser retry loop allows up to three attempts per rubric; transport retries and separately requested metrics add further calls.
A test suite is another level of counting. For SLA’s VLIW task, the upstream submission tests run eight randomized correctness simulations and one cached cycle measurement during a successful full test invocation. Several speed-threshold assertions reuse that measurement. This describes the public benchmark, not a verified copy of SLA’s private evaluator wrapper.
Release coverage. The SLA repository contains the preprint only; the DuoDiscover repository contains its project page and assets. I did not locate official search implementations for Fork-and-Flush, Forgetting, CCTS or MIRA. Their entries therefore describe the paper’s protocol, not code-verified call totals. Benchmark source alone cannot establish a paper’s complete search budget.