benchmark — AI Digest

2 июн Microsoft Build 2026: MAI Model Family Launched to Power GitHub Copilot Without OpenAI Dependency Microsoft models-llm
23 сен Anthropic releases Claude Opus 5.5 at lower prices Anthropic models-llm
7 мая xAI Releases Grok 4.3 with 1M Context, 40-60% Price Cuts, and Agentic Benchmark Gains xAI models-llm
13 мая SenseNova-U1: Open-Source Unified Multimodal Understanding and Generation via NEO-unify SenseTime research
15 мая MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Images Technion research
15 мая EVA-Bench: End-to-End Framework for Evaluating Voice Agents ServiceNow AI research
18 мая ExploitBench: Claude Mythos Preview and GPT-5.5 Develop Real Browser Exploits Autonomously Anthropic research
17 июн VibeThinker-3B Reaches Frontier-Level Reasoning Benchmarks via Curriculum RL WeiboAI research
19 авг AI-Generated Video Attacks on Real-World Crisis Events: RA-Bench shows detectors fail on crisis-media forgery research
19 авг StateM: harness scaling pushes GPT-5.6 Sol to 95.3% on Terminal-Bench 2.1 and the same harness adapts to DeepSeek-V4 Flash for $15 total spend research
26 сен Training Object Permanence in World Models research
23 окт Microsoft's MAI-DxO diagnostic system published in Science Microsoft research
10 мая Google DeepMind's AI Co-Mathematician Reaches 48% on FrontierMath Tier 4 Google DeepMind research
13 мая Baidu Releases ERNIE 5.1 at 6% of Industry Pre-Training Cost, Enters Global Top-10 Search Baidu models-llm
13 мая RubricEM: Meta-RL with Rubric-Guided Policy Decomposition Beyond Verifiable Rewards Google research
18 мая SOOHAK: Frontier LLMs Solve Hard Math But Fail to Recognize Unsolvable Problems research
19 июн ENPIRE: AI Coding Agents Close the Loop on Physical Robotics Research Without Human Intervention NVIDIA / Carnegie Mellon University / UC Berkeley research
26 июн JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting Hao AI Lab, UC San Diego research
3 июл OpenAI Releases GeneBench-Pro, a Frontier Benchmark for AI Agents in Biology OpenAI research
9 июл LingBot-VLA 2.0: Bridging the Gap Between Foundation VLA Models and Real-World Deployment LingBot Team research
11 июл ByteDance EdgeBench: Agent Learning Speed Doubles Every Three Months ByteDance research
30 июл HumanCLAW: Can Vision-Language Models Act Through a Body? Meta Research research
2 авг DeepSeek Puts V4-Flash-0731 API Into Public Beta, Beating Its Own Flagship on Agent Benchmarks DeepSeek models-llm
26 авг GigaChat 3.5 Ultra passes professional-retraining information security exam, scoring 14% above pass threshold Sber research
26 авг Unlocking the Potential of Image Editing via Concept Scaling and Dense Supervision research
26 авг MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks research
28 авг VGI-Bench: Probing Visual Intelligence in Video Generation Models research
23 окт Cohere releases Rerank 5 with Context Focus and calibrated scores Cohere models-llm
23 окт Sber unveils GreenWAM world-action model for robot control Sber research
14 июн MaxProof: MiniMax Model Exceeds IMO and USAMO Gold-Medal Thresholds on Formal Math MiniMax research
4 июл Mistral Releases Leanstral 1.5: Open Formal-Verification Model for Lean 4 Mistral research
8 июл LLM-as-a-Verifier: verification as an independent scaling axis for LLMs Stanford University / UC Berkeley / NVIDIA research
27 авг Apodex 1.1: Scaling Agentic Intelligence for Complex Work Apodex AI research
27 авг ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage research
23 окт Deep Search in Real-World Web Environments research
8 мая AI Co-Mathematician: Google DeepMind Achieves 48% on FrontierMath Tier 4 Google DeepMind research
16 мая MemLens: Benchmark for Multimodal Long-Term Memory in Vision-Language Models NVIDIA research
18 мая Judge Circuits: Mechanistic Explanation of LLM-as-Judge Format Inconsistency research
19 мая CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence (178 HF upvotes) Peking University / Shanghai Artificial Intelligence Laboratory research
19 мая MMSkills: Reusable Multimodal Skills for General Visual Agents (105 HF upvotes) Shanghai Jiao Tong University research
2 июн Crafter: Multi-Agent Harness for Editable Scientific Figure Generation Scores +16pt Over Baselines (103 HF Upvotes) Tsinghua University research
14 июн EvoArena: LLM Agents Score Only 40% on Dynamic Evolving Environments MIT / NUS / Salesforce research
14 июн WeaveBench: Computer-Use Agents Fail at Hybrid GUI+CLI Tasks — 41% Pass Rate Microsoft Research research
17 июн Anthropic Study: Domain Expertise Drives Agentic Coding Success, Not Programming Background Anthropic research
22 июн GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents research
26 июн The Deterministic Horizon: When Extended Reasoning Fails and Tool Delegation Becomes Necessary research
29 июн SingGuard: Runtime Policy-Adaptive Multimodal LLM Guardrail with 56K-Example Benchmark inclusionAI research
3 июл PerceptionRubrics: Atomic Rubric Evaluation Reveals 8% Perception Gap Between Open and Closed Models research
8 июл AutomationBench-AA: 657-task independent benchmark for AI agent SaaS automation Artificial Analysis tools
10 июл Super Weights in LLMs: Why High-Salience Parameters Fail as Fine-Tuning Targets Amazon research
14 июл ABot-AgentOS: General Robotic Agent OS with Lifelong Multi-modal Memory Alibaba research
23 июл Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization research
26 июл Anthropic's Project Pilot tests whether AI models can fly drones Anthropic research
9 авг LongHorizon-Harness proposes Manage-Execute-Audit loop for long-horizon agents research
19 авг Cline open-sources Terminal-Bench-based evals for open-weight coding agents Cline tools
23 авг FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis University of Science and Technology of China research
1 сен Gemini 3.7 Flash agents in Antigravity solve open math problems and build a RISC-V simulator Google DeepMind research
2 сен StudentSim: Training LLM-based Student Simulators Microsoft Research research
8 сен Iris: Climbing to the Search Frontier research
25 сен Schrödinger's Code Repository: have LLMs learned SWE-bench or memorized it? Shanghai Jiao Tong University research
30 апр Programming with Data: test-driven data engineering for self-improving LLMs OpenDataLab research
1 мая AutoResearchBench — a benchmark for autonomous scientific literature search by AI agents BAAI research
8 мая GigaChat Passes Engineering Certification at Moscow Power Engineering Institute Sber industry
11 мая Soohak: 64 Mathematicians Build Research-Level Benchmark That Stumps Frontier LLMs Seoul National University research
12 июн EvoArena: LLM Agents Score Only 39.6% on Dynamic Evolving Environments Benchmark MIT research
25 июн Are We Ready For an Agent-Native Memory System? SJTU Benchmarks 12 Architectures research
6 июл AgenticSTS: Bounded-Memory Testbed for Long-Horizon LLM Agents Alaya Studio research
6 июл EvoPolicyGym: Evaluating Iterative RL Policy Self-Improvement by Coding Agents University of Macau / CUHK research
7 июл SWE-Together: Multi-Turn Benchmark for Coding Agent Evaluation research
10 июл Ideas Have Genomes: Frontier LLMs Score Only 27% on Scientific Lineage Reasoning Shanghai Jiao Tong University research
15 авг LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers University of Illinois Urbana-Champaign research
17 авг DarwinX: Evolving Agent Harnesses Through Natural Selection Salesforce AI Research research
28 авг PAWBench: How Far Are We from Probabilistically Aligned World Modeling? research
12 сен DeepSeek V4.1-Flash tops open-weight evals after GA, at a fraction of Kimi K3 cost DeepSeek models-llm
7 мая Executable World Models for ARC-AGI-3: Coding-Agent Approach Without Game-Specific Logic research
13 мая Learning, Fast and Slow: Dual-Weight Architecture for Continual LLM Adaptation research
8 июн SubtleMemory: Benchmark Reveals Agents Systematically Fail Fine-Grained Relational Memory research
8 июн VideoKR: 315K-Example Training Corpus for Knowledge- and Reasoning-Intensive Video Understanding Yale University research
9 июн SWE-Explore: Benchmarking Repository Exploration as the Binding Constraint in Coding Agents Shanghai Jiao Tong University research
19 июн StylisticBias: 15 Visual Attributes Account for 80% of Social Bias in Multimodal LLMs research
4 июл Will Scaling Improve Social Simulation with LLMs? A Study of 85 Models Stanford / Columbia / Tsinghua research
11 июл UniClawBench: Benchmark for Proactive AI Agents on Real-World Tasks University of Hong Kong research
14 июл AdvancedMathBench: Benchmark Suite for Advanced Mathematical Proof Generation and Verification InternLM research
24 июл GuardianAgentBench: Where Agents Fail and How to Guard Them research
26 июл LLMs Get Lost in Evolving User Intent Microsoft Research research
7 авг MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations Alibaba Group research
8 авг OSReward benchmarks cross-platform computer-use reward models University of Hong Kong research
8 авг GST-Bench tests whether VLMs develop global spatial awareness from video ByteDance Seed research
10 авг GST-Bench: Can VLMs Develop Global Spatial Awareness from Video? ByteDance Seed research
11 авг SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring research
11 авг Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains research
20 авг ASI-Bench: At the Dawn of Artificial Superintelligence 42-author consortium (lead: Junwei Zhou; incl. Chi Wang, Yilun Hao, Yuantao Zhai) research
27 авг FrontierChallenge: Evaluating Scientific Workflow Completion HKU + Apodex AI research
2 сен LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering research