evaluation — AI Digest

15 мая EVA-Bench: End-to-End Framework for Evaluating Voice Agents ServiceNow AI research
7 сен ARC Prize shows GPT-6 Astra's 99.9% ARC-AGI-3 score came from OpenAI's custom harness, not the model OpenAI research
18 мая SOOHAK: Frontier LLMs Solve Hard Math But Fail to Recognize Unsolvable Problems research
19 июн OpenAI Publishes Deployment Simulation: Predicting Model Behavior Before Release OpenAI research
7 июл GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation GigaAI research
13 июл Long-Horizon-Terminal-Bench: Testing Agent Limits on Long-Horizon Terminal Tasks Tencent Hunyuan research
28 авг VGI-Bench: Probing Visual Intelligence in Video Generation Models research
13 сен Specific releases Real-SWE benchmark built from private enterprise codebases Specific tools
15 сен DataFlex-RL: evaluation platform finds RLVR data policies don't beat uniform sampling Peking University research
17 окт Anthropic embeds six METR employees with employee-like access and outlines three-tier external evaluator framework Anthropic industry
24 окт METR and Redwood red-team OpenAI's agent monitoring, find Codex sandbox and Auto-Review bypass METR research
8 июл LLM-as-a-Verifier: verification as an independent scaling axis for LLMs Stanford University / UC Berkeley / NVIDIA research
27 авг ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage research
5 сен Repo-To-Skill: distilling GitHub repositories into reusable AI skills research
5 сен Random Attention: rethinking KV cache eviction for efficient reasoning Salesforce AI Research research
18 сен LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence Stable AI research
23 сен GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay Tencent ARC research
17 окт OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding research
17 окт OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use research
23 окт Deep Search in Real-World Web Environments research
18 мая Judge Circuits: Mechanistic Explanation of LLM-as-Judge Format Inconsistency research
14 июн EvoArena: LLM Agents Score Only 40% on Dynamic Evolving Environments MIT / NUS / Salesforce research
14 июн WeaveBench: Computer-Use Agents Fail at Hybrid GUI+CLI Tasks — 41% Pass Rate Microsoft Research research
3 июл PerceptionRubrics: Atomic Rubric Evaluation Reveals 8% Perception Gap Between Open and Closed Models research
8 июл AutomationBench-AA: 657-task independent benchmark for AI agent SaaS automation Artificial Analysis tools
10 авг OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models NLP Group of The University of Hong Kong research
19 авг Cline open-sources Terminal-Bench-based evals for open-weight coding agents Cline tools
2 сен StudentSim: Training LLM-based Student Simulators Microsoft Research research
5 сен Legibility is not interpretability: judged vs. actual step importance in chain-of-thought research
21 сен Can MiniMax-H3 Reason About the Physical World? research
17 окт FDE-Gym: Benchmarking Agents on Forward-Deployed Engineering Delivery Tasks research
1 мая AutoResearchBench — a benchmark for autonomous scientific literature search by AI agents BAAI research
12 июн EvoArena: LLM Agents Score Only 39.6% on Dynamic Evolving Environments Benchmark MIT research
21 июн FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines Cisco Foundation AI research
6 июл AgenticSTS: Bounded-Memory Testbed for Long-Horizon LLM Agents Alaya Studio research
6 июл EvoPolicyGym: Evaluating Iterative RL Policy Self-Improvement by Coding Agents University of Macau / CUHK research
7 июл SWE-Together: Multi-Turn Benchmark for Coding Agent Evaluation research
10 июл Ideas Have Genomes: Frontier LLMs Score Only 27% on Scientific Lineage Reasoning Shanghai Jiao Tong University research
27 июл K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs research
15 авг How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review research
18 авг HarnessEval-W: Agentifying the Evaluation of Visual Worlds NTU / MirroS-Lab consortium research
28 авг PAWBench: How Far Are We from Probabilistically Aligned World Modeling? research
15 сен Benchmark Radar: a living database and search engine for AI benchmarks Carnegie Mellon University research
19 июн Multimodal Evaluator Preference Collapse: Cross-Modal Contagion in Self-Evolving Agent Loops research
30 июн Formalizing Latent Thoughts: Axiomatic Framework for Evaluating LLM Reasoning Representations University of British Columbia research
4 июл Will Scaling Improve Social Simulation with LLMs? A Study of 85 Models Stanford / Columbia / Tsinghua research
11 июл UniClawBench: Benchmark for Proactive AI Agents on Real-World Tasks University of Hong Kong research
8 авг OSReward benchmarks cross-platform computer-use reward models University of Hong Kong research
19 авг IBM Research benchmarks how much agentic memory each model tier actually needs IBM Research research
27 авг FrontierChallenge: Evaluating Scientific Workflow Completion HKU + Apodex AI research
2 сен xAI publishes LatchBio biosecurity evaluation of Grok 4.6 xAI research
2 сен LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering research
7 сен Last Translation Benchmark: human-authored tasks that still break frontier translation models research
7 сен RealSWE: coding-agent benchmark rebuilt around real, casual user requests research
9 сен Enoki: multi-level hallucination detection from one open IE pipeline Skoltech research
18 сен An Empirical Study of Harness Design for Coding Agents Zoom research
24 окт Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought research