daily notes

Artificial Analysisoriginal post ↗

Today we’re launching our leaderboard for Terminal-Bench-Science 0.1, an agentic benchmark for scientific research work. GPT-6 Astra (max) and Claude Opus 5.5 (xhigh) currently top it at 63% and 62% Terminal-Bench-Science 0.1 was announced in August 2026, built by @StevenDillmann and researchers at Stanford with the @terminalbench team and a global community of open source scientific contributors. It has 70 expert-curated tasks based on real research in five domains: life sciences (19), physical sciences (17), mathematical sciences (17), engineering sciences (9), and earth sciences (8). As with Terminal-Bench, each task drops an agent into a sandbox environment with the data, tools and instructions for a task, and the agent runs from start to finish. Automated tests grade every task pass/fail, and we report the average pass@1 over 3 attempts. Key takeaways: ➤ Only two models score above 50%: GPT-6 Astra (max) at 63%, and Claude Opus 5.5 at 62% (xhigh) and 59% (max). Recent releases have made large advancements, but even the top model, GPT-6 Astra, has significant headroom ➤ Life sciences is the lowest-scoring domain for most leading models, though domain scores are noisy with 8 to 19 tasks each. Claude Opus 5.5 (xhigh) passes 71% of mathematical sciences tasks but 46% of life sciences tasks ➤ The best open weights models, GLM-5.3 (max) at 10% and DeepSeek V4.1 Flash (max) at 9%, sit more than 50 points below the leaders

discussion1 selected reply

ArtificialAnlysreply ↗

Terminal-Bench-Science discriminates well between models, as well as between different reasoning effort levels within frontier models. Claude Opus 5.5 rises ~38 points from low effort at 24% to xhigh at 62%, alongside a 5x difference in the cost per task. Max effort for Opus 5.5 scores slightly below xhigh at 59%. GPT-6 Sol gains 27 points from low to max effort at ~7.5x the cost per task.

The current leaderboard applies a common harness and three-attempt pass@1 average to 70 expert-curated scientific workflows. It is valuable because it exposes the gap between headline model rank, reasoning-effort cost, domain variation, and the remaining open-weight deficit; the linked primary benchmark explains the workflow construction and task-verification standards.

The guest post documents a bounded, independently checked physics task: Claude Science received a simple nine-loop prompt, worked largely unsupervised over days, and produced a result a field expert validated. It also explains why the achievement remains a toy-model calculation using known methods, so the full account is more decision-useful than a headline capability claim.

vllm_projectoriginal post ↗

Great work from the @TileRT_AI and @AIatAMD teams, who got GLM-5.3 to 469 tok/s single-user decode with vLLM on 8× MI355X on @SemiAnalysis_ AgentX. The run uses a disaggregated setup where vLLM handles prefill and TileRT handles latency-critical decode through vLLM's V1 connector interface. 1/2

The reported configuration makes the serving-path decision explicit: use vLLM for prefill and a specialized runtime for latency-critical decode. The 469-token-per-second result is a single-hardware, single-user measurement rather than a model-quality comparison; teams should inspect the public benchmark configuration and measure end-to-end latency before transferring it to production workloads.

This St. Louis Fed study matches 164 industry productivity booms and 174 busts to industry PPI data from 2006 through mid-2025. It finds relative producer-price inflation 3.7 percentage points below the cross-industry average four quarters into a boom, while keeping aggregate AI-inflation effects explicitly unresolved. The methodology, event study, and limits make the full piece more useful than the headline.

GDPNow's September 25 estimate puts Q3 real GDP growth at 5.0%. It is a model-based running estimate using released data rather than an official forecast or an outcome; later releases can still move it.