Times cover one candidate evaluation, excluding the agent’s writing time. ~ marks estimates; hardware alone does not determine the group.
Seconds · up to about a minute
| Task | Benchmark / source | Description & evaluation time | Evaluation resources | Recent progress & saturation |
|---|---|---|---|---|
| Synthetic concept search | CCTS controlled environment | Mutate concept sets and sum utilities. ~Sub-second scoring. | CPU (estimated); simple set operations. | Not applicable. The utility landscape is constructed by CCTS; it has no external research frontier. |
| Causal graph recovery | BNLearn adaptation in MIRA | Score a submitted DAG against the hidden graph. ~1–10 s; discovery and intervention queries are separate. | CPU (estimated); 14–34-node SHD scoring. | Many full-credit runs in this adaptation. MIRA’s results suggest limited headroom on some instances; broader replication remains limited. |
| Symbolic regression | LLM-SRBench subset in MIRA | Score a submitted expression on held-out data. ~1–10 s; fitting/search is additional. | CPU (estimated); numerical expression evaluation. | Still improving. PTB-Search and HarnessSR improve LLM-SRBench results. Near-perfect fit in MIRA does not imply exact equation recovery. |
| VLIW kernel | Anthropic performance take-home | Simulate a submitted instruction schedule. ~1–30 s for correctness and cycle scoring. | CPU; Python simulator, no GPU required. | Headroom beyond paper baselines. Public runs below 1,000 cycles beat the ≈1,112-cycle SLA result; test variants must match. |
| Job-shop scheduling | Frontier-CS #46 | Minimize schedule makespan. ~20–40 s for 15 cases. | CPU; serial case execution assumed. | Agent results available. Recent experiments evaluate this suite; comparison with specialized scheduling methods remains limited. |
| Sphere packing | Frontier-CS #48 | Pack equal spheres into a cube. ~20–40 s for 15 cases. | CPU; serial case execution assumed. | Small recent gains. Agent experiments report improvements on this suite; difficulty varies with the number of spheres. |
| Mobile relay layout | Frontier-CS #307 | Place hubs and assign sensors. ~30–60 s for 10 cases. | CPU; serial case execution assumed. | Narrow score range in one study. Reported results cluster at 72.73–72.82; this alone does not establish saturation. |
Minutes · about 1–10 minutes
| Task | Benchmark / source | Description & evaluation time | Evaluation resources | Recent progress & saturation |
|---|---|---|---|---|
| Circle packing, n = 26 | AlphaEvolve-style task; CCTS | Maximize summed radii. ~Seconds–minutes for a candidate optimization program; checking fixed coordinates alone takes seconds. | CPU (estimated); numerical optimization and geometry checks. | Hard to improve meaningfully. TTT-Discover, ThetaEvolve and SimpleTES repeatedly reach ≈2.635983. Tiny differences require matched feasibility tolerances. |
| Arithmetic Kakeya | CCTS mathematical suite | Optimize an entropy-based construction. ~Seconds–minutes including candidate optimization; a fixed-construction check is cheaper. | CPU (estimated); numerical optimization and entropy scoring. | Diminishing gains at the frontier. The matching four-slope entropy record improved in 2026, but the latest increments are around 10⁻⁹–10⁻¹¹. |
| Heilbronn’s triangle | CCTS mathematical suite | Place 11 points to maximize the smallest triangle. ~Seconds–minutes including candidate optimization. | CPU (estimated); optimization and geometric checks. | Hard to improve at n = 11. MLEvolve and AlphaEvolve effectively tie at 0.03652989; the Arena leaders reproduce it. |
| Squares in a square | CCTS mathematical suite | Maximize the summed side length of non-overlapping squares. ~Seconds–minutes including candidate optimization. | CPU (estimated); optimization and overlap checks. | Depends on the instance size. Recent theory studies the same side-length objective; empirical comparisons for CCTS’s exact task remain limited. |
| Order-add | EdgeBench | Score a permutation through a cost oracle. ~Seconds–minutes per score; extended solver runs can take much longer. | CPU (estimated); permutation oracle. | Grader issue affects comparisons. An independent audit finds best-submission selection errors that can hide later improvements. |
| LLM-SQL cache optimization | Frontier-CS research track | Reorder data to improve prefix-cache reuse. ~10 s–3 min for deterministic scoring. | CPU (estimated); no LLM inference in the scorer. | Plateau within one study. Several methods share a 27.05 best score; broader evidence of saturation is limited. |
| Lasso path | TTT-Discover | Compile and verify a regularization-path solver on 17 cases. ~Seconds–minutes overall. Reference generation and checking add to solver latency. | One CPU thread; AMD EPYC 9745. | Recent gains remain. Evolving in Thought Space. |
| Physics discovery | DiscoverPhysics adaptation in MIRA | Score a law on held-out trajectories and grade its explanation. ~Seconds–minutes, including the final judge call. | CPU simulation (estimated) + hosted GPT-5.5 judge. | Not broadly saturated. The original benchmark still has unsolved worlds; MIRA uses an adaptation, so scores cannot be pooled. |
| Olympiad proof generation | IMOProofBench Advanced | Grade a submitted proof. ~Tens of seconds–minutes per model judgment. | LLM judge + CPU harness; hosted-model hardware is not reported. | Saturated for strong-agent solve rate. MIRA and GitSwarm both report all 30 solved with GPT-5.5 under their grading protocols. |
| Research proposal generation | Svalbard Idea Vault / IdeaScientist | Assess a final proposal with five rubric calls. ~Minutes overall; review latency and retries vary. No proposed experiment is executed. | LLM judge API + CPU harness; five logical rubric calls. | New evaluation; limited follow-up. IdeaScientist introduces this task; proposal ratings do not measure experimental success. |
| Polyomino packing | Frontier-CS #0 | Minimize the packing rectangle. ~2–3 min for 70 cases. | CPU; serial case execution assumed. | Still improving. Test-Time Communication reports a new best-known result; Fork-and-Flush finds further search gains. Evolving in Thought Space also advances the frontier. |
| Inference kernels | TPU-Inference | Optimize inference kernels. 3 min typical; 1–8 min across the suite. | TPU v6e; chip count not specified. | Recent gains over tuned baselines. DuoDiscover improves tuned Pallas kernels on this suite. |
| Attention kernels | Flash Attention 4 | Optimize attention configurations. 4 min for full evaluation. | 1× NVIDIA B200. | Still improving despite mature baselines. AVO beats FA4 by up to 10.5% on selected B200 shapes; DuoDiscover also finds gains. |
| Apple production planning | AtCoder AHC058 | Allocate machine investments. ~5 min serial for 150 × 2 s cases. Parallel cases shorten wall time. | CPU; eight case workers per candidate in the reported run. | Small frontier gains. Evolving in Thought Space. |
| TriMul | TTT-Discover / GPUMode | Compile, check and time triangle multiplication. ~Minutes; 520 s evaluator timeout. | 1× H100 80 GB per verifier. | Narrow frontier gaps. Evolving in Thought Space. |
| nanochat training | nanochat-autoresearch | Train and validate a small language model. ~5–7 min including the fixed five-minute training run. | 1× H100. | Still improving. Forgetting, GitSwarm and rekursiv.ai report gains; H100 and B200 scores need separate comparisons. |
Long evaluations · 10 minutes or more
| Task | Benchmark / source | Description & evaluation time | Evaluation resources | Recent progress & saturation |
|---|---|---|---|---|
| AHC014 / RectJoin | AtCoder; Frontier-CS #159 | Add valid rectangles to a dot grid. ~10–15 min for 75 cases; the 30-case variant is ~1–5 min. | CPU; serial case execution assumed. | Recent agent gains. Fork-and-Flush and Forgetting improve agent baselines; case sets and budgets differ. |
| Ice-rink routing | Frontier-CS #171 | Visit targets using movement and block edits. ~10–15 min for 75 cases. | CPU; serial case execution assumed. | Recent agent gains. Forgetting and Fork-and-Flush improve baselines; 100 is a reference score, not a ceiling. |
| Database execution | SQLite case study | Run SQLite correctness and performance workloads. About 10 min (reported). | 44-vCPU node. | Recent workload gains. DuoDiscover improves SQLite performance on its tested workload. |
| GPU kernels | SOL-ExecBench | Compile, verify and time SOL-Exec kernels. 14 min typical; 2–60 min in DuoDiscover’s full evaluator. Smaller SLA cases can be faster. | 1× B200 per kernel for these timings. | Still improving overall. SLA and DuoDiscover find gains. Difficulty varies by kernel; a Triton plateau is not a hardware ceiling. |
| Workload speedups | SWE-fficiency-Lite | Run repository correctness and performance workloads. 15 min typical; 3–93 min. | 16 vCPUs, 32 GB RAM per evaluation node. | Still improving. PerfAgent raises expert-matching patches from 26% to 74%; DuoDiscover reports further workload speedups. |
| Toroidal string placement | AHC004; Frontier-CS #150 | Fit strings into a wrapping grid. ~15–20 min for 100 cases. | CPU; serial case execution assumed. | Recent agent gains. Search improves agent solutions; comparisons with the strongest contest solutions require matched limits. |
| JAX kernels | JAXBench | Compile, verify and time TPU kernels. 17 min typical; 7–32 min. | TPU v6e; chip count not specified. | Still improving. JAXBench leaves gaps to tuned Pallas kernels; DuoDiscover reports newer gains. Hardware and subsets differ. |
| Compression | zstd case study | Measure zstd while requiring identical output bytes. 15–25 min. | 44-vCPU node. | Recent workload gains. DuoDiscover improves byte-preserving zstd execution; broader codec results optimize different objectives. |
| libexpat → assembly; Git → Zig; Dart → Haskell; Lua compiler | FrontierSWE | Build and test a software implementation or port. ~1–30 min, depending on the suite. | CPU-only tasks; core count and RAM not reported. | Mixed. SLA brings Lua near full coverage; FrontierSWE V2 changes and expands the suite, so old scores do not establish broad saturation. |
| CPU architecture search | HillClimbBench-CPU | Train each candidate architecture from scratch. ~Minutes–tens of minutes per trial. | One CPU thread. | Limited independent evidence. HillClimbBench-CPU is a custom task distribution with few independent results. |
| Offloaded MoE inference | FreeToken case study | Benchmark inference with weights offloaded to host memory. About 70 min (reported). | 1× L40S 48 GB + 433 GB host-resident weights; additional RAM overhead needed. | Workload-specific results. DuoDiscover studies one FreeToken configuration; hardware and model choices affect comparisons. |
| LLM serving | vLLM case study | Run a serving-load sweep and correctness tests. About 85 min (reported). | 8× B200 node. | Substantial gains on the tested workload. DuoDiscover improves serving performance; results depend on the vLLM version and traffic mix. |
| Borden source inversion | EdgeBench | Run forward simulation and recover a contamination source. ~Minutes–hours per candidate solver. | CPU numerical simulation (estimated); parallelism depends on the solver. | Formulation affects score interpretation. A recent audit finds unstated physics conventions. Score gains can reflect learning the checker rather than better inversion. |
| tinyCIFAR | Fork-and-Flush custom task | Train a classifier and score accuracy against compressed size. ~Minutes–hours per training trial. | GPU training; device and count not reported. | Limited independent evidence. This custom accuracy–size objective has little independent replication; ordinary CIFAR accuracy records measure a different objective. |
| ML pipeline development | MLE-bench (75 Kaggle competitions) | Train a candidate pipeline and score its predictions. ~Minutes–hours per trial (estimated; dataset-dependent); scoring existing predictions alone is cheaper. | RSI-top: 1× A100, with a 12-hour total search budget, not a 12-hour evaluation. | Headroom remains across the suite. RSI-top reports a 63.1% mean medal rate; a medal is a competition-relative threshold, not an upper bound on solution quality. |
| RMT / Loop Transformer design | HillClimbBench-RMT / -Loop | Train candidate architectures and measure held-out loss. ~Minutes–hours per trial. | Training compute; device/count not reported. Timing estimate assumes GPU training. | Still improving in recent studies. Both MIRA and GitSwarm improve starting designs; compare training budgets before ranking results. |
| Structured-LWE recovery | Frontier-CS 2.0 | Run a candidate solver and verify recovered secrets. ~Minutes–hours or longer; checking an existing witness alone is much cheaper. | CPU lattice/linear-algebra computation (estimated); solver memory needs vary. | Unsolved instances remain. Fork-and-Flush and SLA leave instances unsolved; comparisons with specialized attacks remain limited. |
| Repository speedups | FormulaCode-V | Run full scientific-Python workloads. 7 min–6.8 h, depending on the repository. | 2 vCPUs, 4 GB RAM per evaluation node. | Still improving. FormulaCode exposes unresolved expert gaps; DuoDiscover improves a selected subset. Their aggregate scores are not interchangeable. |
| CPU data prefetching | DPC4 / ChampSim | Simulate a prefetcher across workloads. Up to 28 h wall time for the full evaluation. | ≈2,600 CPU-hours; ≤120 concurrent simulations per configuration. | Still improving, with costly gains. ArchAgent v2 beats the DPC4 champion by 0.3% overall; DuoDiscover explores further designs. |
Timing, hardware and grouping notes
Groups use the reported typical time where available. For broad ranges without a typical value, placement uses the slower end of the estimate; the full range remains visible. They are browsing categories, not fixed runtime guarantees.
Resource estimates describe a plausible evaluation setup, not hardware reported by the paper. A CPU core count or GPU count is stated only when supported by the cited protocol. Resources exclude the model that proposes candidates; model-judge calls are part of evaluation.
DuoDiscover’s Table 8 and hardware description pair its measured times with evaluation hardware. Its CPU cluster capacity is not the cost of one prefetcher evaluation; that row gives per-evaluation CPU-hours and simulation concurrency. Kernel latency alone also excludes compilation and correctness checks.
For mathematical construction tasks, the minute-scale estimate includes running a candidate optimization program. Checking an already-produced construction is faster. Graph and expression rows instead score already-submitted artifacts.
For the Frontier-CS suites, a useful budget is number of cases × solver time per case, plus compilation and checking. The public configs give polyomino 70 × 2 s, job-shop and sphere packing 15 × 1 s, relay layout 10 × 4 s, AHC014 and p171 75 × 10 s, and p150 100 × 10 s. These estimate a solver that uses most of its allowance, not an observed average. Faster solvers, fewer cases and parallel workers reduce wall time; the papers may use different subsets.
Other ranges are order-of-magnitude judgments from the task: direct numerical checks are cheap, while compilation, repeated test workloads, simulations and retraining add cost. Unbounded numerical search has no fixed duration. MIRA’s task protocols, for example, distinguish graph scoring from training each architecture afresh. Model-judged tasks include the evaluator’s LLM calls. A larger judge or a slower service can push them beyond the range shown.