inference — AI Digest

18 авг Stripe to acquire OpenRouter for over $7B, putting AI model routing and payments under one roof industry
26 авг OpenAI unveils Jalapeño, a Broadcom-built 700W inference ASIC that beats Nvidia Blackwell on tokens-per-watt OpenAI industry
7 мая xAI Releases Grok 4.3 with 1M Context, 40-60% Price Cuts, and Agentic Benchmark Gains xAI models-llm
13 мая Codex-Spark (GPT-5.3-Codex-Spark) Research Preview: 1000+ Tokens/Second Coding Model OpenAI tools
20 мая Gemini 3.5 Flash Released at Google I/O 2026: Frontier Coding + Agentic at Flash Speed Google DeepMind models-llm
8 июн NVIDIA Nemotron 3 Ultra: Open 550B MoE Model Now Available for Agentic Workloads NVIDIA models-llm
25 июн OpenAI and Broadcom Unveil Jalapeño: OpenAI's First Custom AI Inference Chip OpenAI industry
25 июн Qualcomm Acquires Modular for $3.92B to Challenge CUDA Lock-in Qualcomm industry
13 июл vLLM v0.25.0: Model Runner V2 Default, PagedAttention Retired, Transformers Backend Parity tools
9 мая Zyphra Releases ZAYA1-8B: Open Reasoning MoE Model Trained on AMD Hardware Zyphra models-llm
4 июн JetBrains Open-Sources Mellum2: 12B MoE Coding Model for Multi-Model Pipelines JetBrains models-llm
16 июн Kimi K2.7-Code HighSpeed: 6× Throughput for Production Coding Agent Pipelines Moonshot AI models-llm
19 июн AWS Summit New York 2026: Bedrock AgentCore GA, Kiro iOS Preview, and AWS Context Previewed Amazon tools
24 июн Mistral Releases OCR 4 with Bounding Boxes, Block Classification, and 170-Language Support Mistral tools
26 июн JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting Hao AI Lab, UC San Diego research
29 июн DeepSeek Open-Sources DSpark: 57–85% Inference Speedup for V4 in Production DeepSeek tools
29 июн JetSpec: Parallel Tree Drafting Achieves 9.64× Speculative Decoding Speedup Hao AI Lab, UCSD research
6 июл Program-as-Weights: Compiling Task Specs into LoRA Adapters for On-Device Inference University of Waterloo / Harvard research
6 июл MrFlow: Training-Free 10-25x Speedup for Flow-Matching Text-to-Image Models Beihang University research
16 июл PrismML Releases Bonsai 27B: First 27B-Class Model to Run on iPhone PrismML models-llm
19 мая LongLive-2.0: NVFP4 Parallel Infrastructure for Long Video Generation (NVIDIA, 1,220 HF upvotes) NVIDIA research
14 июн MiniMax Sparse Attention: 28× Compute Reduction at 1M-Token Context with No Quality Loss MiniMax research
28 авг vLLM v0.28.0 ships Kimi-K3 perf push, full DeepSeek-V4 sparse-MLA, Model Runner V2 maturity, and tiered KV-cache offloading vLLM Project tools
9 сен Uno: lossless LLM speedups via diffusion-augmented autoregression MBZUAI research
6 мая SGLang v0.5.11: Speculative Decoding V2 as Default and Eight New Model Architectures tools
10 мая vLLM v0.20.2: TurboQuant 2-bit KV Cache and FlashAttention 4 Default for MoE Serving tools
15 мая Hugging Face Transformers: Async Continuous Batching Achieves 22% Inference Speedup Hugging Face tools
16 мая Orthrus: 7.8x Inference Speedup for Qwen3 via Autoregressive-Diffusion KV Sharing research
18 мая vLLM v0.21.0: Blackwell MLA Backend, HMA KV Offload, Spec Decode for Reasoning Models vLLM Project tools
2 июн vLLM v0.22.0: DeepSeek V4 Production Hardening, Rust Frontend, 28.9% Latency Drop tools
9 июн vLLM Semantic Router v0.3 Themis: Stateful Production Routing with Session-Aware Agentic Routing tools
14 июн vLLM Adds Day-0 Support for MiniMax M3 Open Weights with 1M-Context Sparse Attention MiniMax tools
17 июн vLLM v0.23.0: Model Runner V2 Default for Llama and Mistral, Transformers v5, Multi-Tier KV Cache tools
2 июл vLLM v0.24.0: Model Runner V2 Default, Rust Frontend, SM90 FP8 Speedups vLLM tools
3 июл Program-as-Weights: Compile-Once Adapter Paradigm Matches 32B Models at 1/50 the Memory research
11 авг vLLM v0.27.0 Ships Kimi K3 Support and DeepSeek-V4 Optimizations vLLM Project tools
26 авг llama.cpp v0.3.0 — first tagged stable release of the b10621 nightly; dots3-note multimodal, GLM-4.5-Air MTP, DeepSeek 4 tensor-split, ggml v0.22.0 ggml-org tools
5 сен llama.cpp v0.4.0: lazy tensor reading, per-slot context limits, video input tools
18 сен DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression DeepSeek research
23 окт Native PyTorch TPU engines ship for vLLM and SGLang Google tools
24 окт Modular makes the MAX inference stack source-available under a permissive license Modular tools
29 апр vLLM v0.20.0 — third release in two weeks vLLM tools
30 апр TIDE: cross-architecture distillation for diffusion LLMs Peking University research
2 мая ESamp: LLMs explore by latent distilling for semantic-novelty sampling ShanghaiTech University research
11 мая AutoTTS: LLM Agents Automatically Discover Test-Time Scaling Strategies for $40 research
2 июн BadHost (CVE-2026-48710): Host-Header Auth Bypass in Starlette Exposes vLLM, LiteLLM, and MCP Servers tools
29 июн Wayfinder Router: Open-Source Offline LLM Query Router Trends on Hacker News tools
2 июл BlockPilot: Instance-Adaptive Block Size for Diffusion-Based Speculative Decoding research
18 сен Infinite-Parameter LLMs: arXiv paper proposes generating weights from live data Independent researchers research
6 сен DeepSeek plans a 160,000-chip Huawei Ascend cluster in Inner Mongolia DeepSeek industry
4 мая Ollama v0.23.0 Adds Claude Desktop Support via Ollama Launch Ollama tools
4 мая vLLM v0.20.1 Patches Critical DeepSeek V4 Instability Under Production Workloads vLLM Project tools
6 мая vLLM v0.20.1: DeepSeek V4 Stabilization on CUDA 13 and PyTorch 2.11 tools
6 мая Ollama v0.23.1: Gemma 4 MTP Speculative Decoding Delivers 2× Speed on Apple Silicon tools
7 мая VK Video AI Character Recognition Boosts Watch Time 9% via Cascade Face Detection VK AI tools
7 мая LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents Shanghai Jiao Tong University research
9 мая llama.cpp b9085: MiMo-V2.5 Flash Attention and Vertex AI Server Support tools
12 мая vLLM v0.21.0rc1: Python 3.14, CUDA 13.0, and Transformers v5 Compatibility tools
12 мая TMAS: Scaling Test-Time Compute via Multi-Agent Synergy with Hierarchical Memory research
13 мая vLLM v0.21.0rc1: PyTorch 2.11, HuggingFace Transformers v5, and Python 3.14 Support tools
15 мая Ollama v0.24.0: Codex App Integration and MLX Sampler Improvements Ollama tools
16 мая llama.cpp b9161/b9169: Codex CLI Compatibility and Qwen3A Multimodal Support ggml-org tools
18 мая BetaPRM: Uncertainty-Aware Process Rewards Cut Reasoning Token Use by 33% research
8 июн Code2LoRA: Hypernetwork Generates Repo-Specific Adapters for Code LMs with Zero Inference Overhead University of Waterloo research
9 июн Ollama v0.30.7: Hermes Desktop Support, Gemma 4 QAT, and Nemotron-3-Ultra Ollama tools
11 июн llama.cpp b9589–b9592: CUDA SSM Sync Fix and Mamba Memory Optimization tools
12 июн llama.cpp b9603: Qualcomm Adreno OpenCL Kernels for On-Device Inference ggml-org tools
17 июн Ollama v0.30.9: Cohere2Moe Support, Coding Agent Single-Token Output Bug Fixed tools
17 июн llama.cpp June 16 Builds: Eagle3 Speculative Decoding, Vulkan UMA Memory, NVFP4 Fixes tools
19 июн llama.cpp b9716 Builds: InternVL Multimodal Batching, CUDA col2im, and Nginx SSE Fix tools
22 июн llama.cpp b9754: Real-Time Model Load Progress via SSE and PEG Grammar Parser tools
24 июн Modal Launches Auto Endpoints for Production-Grade Open-Model LLM Inference Modal tools
29 июн llama.cpp Builds b9830–b9837: DFlash v2, MiniCPM5 Parser, --reasoning-preserve Flag ggml-org tools
2 июл Ollama v0.31.1: Gemma 4 Nearly 90% Faster on Apple Silicon via MTP Ollama tools
3 июл ELDR: Expert-Locality-Aware Routing Cuts MoE Serving Latency by up to 14% Microsoft Research research
8 июл Ollama v0.31.2: MLX small-batch matmul kernel, llama.cpp build 9840, CUDA updates tools
12 июл llama.cpp b9967–b9969: Adreno GPU Acceleration and OpenAI-Compatible Null Sampling tools
14 июл llama.cpp Adds Tencent Hunyuan 3, Minimax2 Eagle3 Speculative Decoding, and SYCL Fused MoE tools
23 июл vLLM previews production-scale support for Moonshot AI's Kimi K3 ahead of its weight release vLLM / Moonshot AI tools
19 авг Dharma-AI lifts GPU-cluster utilization 53.6% → 87.0% by encoding physical constraints into the scheduler Dharma-AI tools
19 авг llama.cpp ships v0.1.2 pre-release plus daily b10483–b10488 with SYCL perf fix, OpenVINO bump, DGX Spark CUDA tuning llama.cpp (GGML) tools
19 авг Groq closes $350M Series A to build an AI inference cloud Groq industry
23 авг SGLang v0.5.18 lands major model and perf updates tools
23 авг llama.cpp rolls up Aug 22 backend and model fixes tools
25 авг llama.cpp b10603-b10615 — GLM-4.5-Air MTP, Deepseek 4 -sm tensor, Metal flash-attn vec tuning for M1 Pro/M2 Ultra/M5 Max ggml-org tools
25 авг SGLang v0.5.18 SGLang tools
29 авг llama.cpp nightly burst b10668-b10679 includes Vulkan wrong-token fix llama.cpp tools
1 сен llama.cpp brings MTP speculative decoding to recurrent Qwen models ggml tools
2 сен llama.cpp Sep 1-2 wave: +4.9% generation from n-gram lookup fix, fused CUDA MoE reduction ggml tools
6 сен SGLang v0.5.19 lands beam search, DeepEP v2, and Qwen3.8 support across 786 PRs tools
8 сен Language Models Can Control Their Own Attention research
8 сен Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference Cerebras research
10 сен vLLM v0.29.0 makes Model Runner V2 the default and adds Mamba prefix caching vLLM tools
15 сен llama.cpp v0.4.1 adds Maple 20B, Tencent Hy 4, and Spark2.5 support ggml tools
16 сен Ollama v0.34.1 graduates MLX safetensors model creation and speeds up /api/tags tenfold Ollama tools
17 сен llama.cpp patches remotely exploitable use-after-free in llama-server RPC endpoint ggml tools
23 сен vLLM v0.30.0 published: DeepSeek-V4.1-Flash with MXFP8 FlashMLA KV on Blackwell vLLM project tools
24 сен llama.cpp v0.5.0 stable: CUDA/Metal kernel work, video_url in server tools
17 окт Tencent opens up DSpine parallel speculative-decoding drafter Tencent research
17 окт Ollama v0.40.5 adds NVIDIA Nemotron 3.5 Super and cloud decision models Ollama tools
23 окт vLLM 0.32.0 widens low-precision KV cache and drops Python 3.10 vLLM tools
23 окт SGLang 0.5.22 makes Rust prefix cache default and adds Apple Silicon LMSYS tools
24 окт vLLM serves 1M-token GLM-5.3 contexts on 16 H200s with PCP, DCP and Hybrid HiSparse vLLM tools
12 мая llama.cpp Adds gpt-oss-20b Support in May 12 Build tools
13 мая Ollama v0.23.3: MLX Runner Fixes and macOS 26 Metal Compatibility Ollama tools
22 сен vLLM v0.30.0 tagged on GitHub; release not yet published to PyPI vLLM project tools
24 сен Ollama v0.34.4 pre-release fixes model-not-found flake, thinking-model structured outputs tools