key developments

willison highlights the “headless everything” trend as saas shifts toward api-first for ai agents. simon willison flagged a convergence of signals: matt webb arguing headless services will dominate because personal ais prefer apis over guis, salesforce announcing “headless 360” exposing its entire platform as apis/mcp/cli, and brandur leach predicting a second wave of the api-first economy. the thesis is straightforward: if agents become the primary interface to services, then the gui becomes vestigial and the api becomes the product. this matters because it threatens per-seat saas pricing models (agents don’t have “heads” to count) and creates a selection pressure where services without robust apis lose to those that have them. willison draws the parallel to the early 2010s api boom, but the difference now is that the consumer of the api is autonomous software, not a developer building an integration. companies that gated their apis or let them rot over the past decade are suddenly exposed. https://simonwillison.net/2026/Apr/19/headless-everything/#atom-everything

mixture-of-depths attention (moda) offers a promising new primitive for scaling transformer depth efficiently. a new paper introduces moda, which lets attention heads attend not just to kv pairs at the current layer but also to “depth kv pairs” from preceding layers. this directly addresses signal degradation in deep transformers, where useful features from early layers get diluted through repeated residual updates. at 1.5b parameters, moda improves average perplexity by 0.2 across 10 validation benchmarks and boosts downstream task performance by 2.11%, with only 3.7% flops overhead. critically, they also provide a hardware-efficient implementation achieving 97.3% of flashattention-2’s throughput at 64k sequence length, which means this isn’t just a theoretical contribution. the finding that moda works better with post-norm than pre-norm is a useful architectural detail. this is the kind of architectural primitive that could compound in value as models scale deeper. https://arxiv.org/abs/2603.15619

llama.cpp merges speculative checkpointing with significant but variable speedups. speculative checkpointing (pr #19493) landed in llama.cpp, using ngram-based draft speculation to speed up inference. results vary substantially by model and task: one user reports 665% speedup on devstral small for minor code edits, while qwen 3.6 only gains ~40% (improved to 140% with tuned params). the variability is the important takeaway; this is not a universal win but a powerful optimization for repetitive or predictable generation patterns, which code editing often is. the community is actively tuning parameters (ngram size, draft min/max, repeat penalty) per model, suggesting this will take time to mature into reliable defaults. https://github.com/ggml-org/llama.cpp/pull/19493

notable

  • programasweights (paw) compiles english function descriptions into 22mb neural programs; a paw-adapted 0.6b model beats raw-prompted qwen3 32b on fuzzy tasks. clever approach where a compiler (finetuned qwen3-4b) outputs lora weights in a single forward pass, no gradient descent at compile time. interesting for edge deployment. https://www.reddit.com/r/MachineLearning/comments/1spqcze/

  • cloudflare’s “unweight” achieves 15-22% lossless compression of llm weights by huffman-coding the exponent byte of bf16, with on-chip decompression to preserve bandwidth gains. bit-exact outputs, no quality loss, saves ~3gb vram on 8b models. elegant exploitation of the fact that 99% of exponents in a layer cluster into just 16 values. https://www.reddit.com/r/LocalLLaMA/comments/1spnx1l/

  • streamforge claims to run 40gb models on 3gb vram by streaming one transformer block at a time with async dma prefetch; 130 lines of python, full bf16, no quantization. the approach is straightforward (sequential block execution with pipelined transfers) but the implementation being this compact is notable for local inference enthusiasts. https://github.com/madtunebk/streamforge

  • the sequence radar previews a series on transformer alternatives (text diffusion models, ssms) and notes the frontier is splitting into generalist reasoning models, domain specialists, and workflow-native agents. useful framing of the current product landscape. https://thesequence.substack.com/p/the-sequence-radar-845-last-week

  • 1,200 iclr 2026 papers (~22% of accepted) now have public code/data links available ahead of the conference starting april 22 in rio. useful resource for anyone tracking the research landscape. https://www.paperdigest.org/2026/04/iclr-2026-papers-with-code-data/

  • independent replication finds llms appear to think in geometry, not language, in middle transformer layers. confirms recent findings (arxiv 2411.04986, 2411.08745) via alternative methodology; the middle layers operate in a language-independent geometric space while early/late layers handle linguistic i/o. https://www.reddit.com/r/LocalLLaMA/comments/1spy497/

papers

“mixture-of-depths attention”, zhu et al. 2026. introduces moda, enabling attention heads to attend across depth (preceding layers’ kv pairs), addressing signal degradation in deep transformers with minimal flops overhead and near-flashattention-2 efficiency. https://arxiv.org/abs/2603.15619

“test-time scaling makes overtraining compute-optimal”, roberts et al. 2026. title alone signals an important result for training/inference compute allocation decisions; connects overtraining regimes with test-time compute scaling. https://www.reddit.com/r/mlscaling/comments/1spww4f/