09 September 2026

20 papers shortlisted this week — most discussed: Compile by Training: Turning Natural-Language Specifications into Loca (378 upvotes).

One week of arXiv Picks, 6 stories, as published.

Systems & Inference

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

This work challenges the standard paradigm of KV cache compression, which relies on complex scoring metrics to evict less important tokens. The authors demonstrate that the selection signal in these methods contributes almost nothing, and that a simple random eviction strategy performs surprisingly well.

Why it matters — Complex, computationally expensive KV cache eviction algorithms can be replaced with simple random eviction without significant loss in reasoning performance, drastically simplifying inference engine design.

arXiv · 09 September 2026 · Read the original →

Training & Efficiency

Rethinking On-Policy Distillation of Large Language Models II: One Training Example

This study explores the data limits of on-policy distillation (OPD) by training a student model using only a single query. Surprisingly, this one-shot distillation continues to improve the student model over hundreds of steps, recovering the majority of the performance gains achieved by full-dataset distillation.

Why it matters — On-policy distillation does not require massive datasets; training on even a single query can yield continuous optimization gains and recover most of the performance of full-data distillation.

arXiv · 09 September 2026 · Read the original →

Systems & Inference

Compile by Training: Turning Natural-Language Specifications into Local Neural Functions

This paper introduces a method to compile natural-language specifications into local, reusable neural functions. It uses remote teacher models to generate task-specific training examples, which are then used to train a local student model, eliminating the latency, cost, and dependency of calling remote APIs.

Why it matters — Engineers can replace expensive, high-latency remote API calls for structured text tasks with local, specialized neural functions trained on synthetic data generated from a single natural-language specification.

arXiv · 09 September 2026 · Read the original →

Agents & Tools

Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments

The authors present a framework that converts static agent trajectories into executable, scalable terminal environments. This allows agent post-training to benefit from interactive execution feedback and multiple verifiable tasks, rather than relying on frozen, single-use demonstrations.

Why it matters — Instead of manually building complex interactive environments for agent training, developers can programmatically synthesize executable environments directly from existing agent execution logs.

arXiv · 09 September 2026 · Read the original →

Systems & Inference

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

The paper introduces diffusion-augmented LLMs, which combine autoregressive model distributions with discrete diffusion to generate multiple tokens in parallel. This approach decouples the parallel generation process from the autoregressive distribution to achieve lossless speedups over standard sequential generation.

Why it matters — It demonstrates a method to bypass the sequential autoregressive bottleneck by using discrete diffusion to generate multiple tokens in parallel without altering the underlying model distribution.

arXiv · 09 September 2026 · Read the original →

LLMs & Reasoning

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

This paper addresses the fragility of self-improvement in reasoning models, where sparse terminal verifiers and overconfident dense self-guidance can lead to narrow or incorrect solution paths. The proposed FlowBalance method stabilizes learning by training a normalized distribution over complete reasoning paths grounded by verifiers.

Why it matters — It provides a method to stabilize RL-based self-improvement in reasoning models by balancing sparse terminal rewards with a normalized distribution over complete reasoning paths, preventing the model from overconcentrating on narrow solution modes.

arXiv · 09 September 2026 · Read the original →

Another week of trying to optimize our way out of the hardware bill.

6 stories, every Wednesday

Published here every week. Follow by RSS to get it as it lands.

Subscribe by RSS All issues

← Previous issue Next issue →