PJFP.com

Pursuit of Joy, Fulfillment, and Purpose

Tag: GPQA Diamond

  • Inkling: Thinking Machines Lab Releases Its First Open-Weights Model, a 975B Multimodal Mixture-of-Experts With Controllable Thinking Effort That Can Fine-Tune Itself on Tinker

    Thinking Machines Lab, the AI startup founded by former OpenAI CTO Mira Murati, has released Inkling, its first open-weights model trained from scratch. Inkling is a 975 billion parameter Mixture-of-Experts transformer (41B active) with a context window of up to 1 million tokens, native multimodal reasoning over text, images, and audio, and a dial for controllable thinking effort. The lab is explicit that Inkling is not the strongest model in the world. It is pitched as something arguably more useful: a broad, balanced, customizable foundation you can fine-tune on Tinker, with the full weights on Hugging Face. The announcement even includes a demo where Inkling fine-tunes itself and swaps in its own new weights.

    TLDR

    Thinking Machines Lab released Inkling, a 975B-total, 41B-active Mixture-of-Experts model pretrained on 45 trillion tokens of text, images, audio, and video, alongside a preview of Inkling-Small (276B total, 12B active). The release covers the model’s generalist benchmark profile across reasoning, agentic coding, tool use, vision, and audio; a controllable thinking effort setting that lets developers trade performance against tokens (matching Nemotron 3 Ultra on Terminal Bench 2.1 at roughly a third of the tokens); an encoder-free multimodal architecture using dMel spectrograms and hMLP image patches; a training recipe combining Muon and Adam with weight decay coupled to the learning rate; RL scaled past 30 million rollouts with log-linearly improving reasoning and an emergent compression of the chain of thought; an epistemics push covering calibration, forecasting (where it beats several frontier models), abstention, and censorship resistance; the strongest FORTRESS adversarial safety score among compared open-weights models; a headline-grabbing demo of the model fine-tuning itself into a lipogram assistant via Tinker; and day-one availability on Tinker (at a 50% discount), Hugging Face, and inference partners including Together, Fireworks, Modal, Databricks, Baseten, vLLM, SGLang, and llama.cpp.

    Thoughts

    The most striking thing about this launch is its honesty. Nearly every frontier release leads with a claim to be the best at something, and the fine print walks it back. Thinking Machines Lab says plainly that Inkling is not the strongest model available, open or closed, and then makes the case that “strongest” is the wrong axis for most real buyers. If you are going to run a model millions of times inside a product, what you care about is the cost curve, the adaptability, and whether you can shape it to your workflow. That framing conveniently matches their business (Tinker sells fine-tuning), but it also matches how production AI actually gets deployed, where cost and latency are binding constraints and a benchmark crown is trivia.

    The self-fine-tuning demo deserves more attention than it will probably get. Asked to become a lipogram assistant that never uses the letter “e” (a behavior prompting alone cannot reliably produce), Inkling wrote its own training objective and scoring function, generated its own synthetic data, launched the run on Tinker, evaluated the result against its base self, and then staged a weight swap so the improved checkpoint took over the session. That is a closed loop of specify, train, evaluate, and self-update, packaged as a cute product demo. The loop is the primitive behind every serious conversation about recursive self-improvement, and here it is running as a marketing asset with a 27 minute wall clock. The gap between “toy objective” and “economically meaningful objective” is now a question of reward design, not plumbing.

    Controllable thinking effort is the feature I expect developers to care about most. Instead of publishing a single score, TML publishes a curve: sweep the effort setting from 0.2 to 0.99 and watch performance trade against generated tokens. Inkling reportedly matches Nemotron 3 Ultra on Terminal Bench 2.1 while spending about a third of the tokens. Benchmarks reported as single points hide exactly this, and a model that reaches a target score cheaply beats a model that scores two points higher at triple the cost in any high-volume workload. Expect effort curves to become standard marketing for open models, the way context length became standard a couple of years ago.

    The epistemics section is quietly the most differentiated part of the release. TML trained calibration directly, running RL against proper scoring rules on resolved real-world questions, and pairing a rubric grader with a claims grader that does agentic web search to verify each factual assertion. The result is a model that beats GPT-5.5 and Claude Opus 4.8 on ForecastBench without search and holds its own on Prophet Arena. A model that knows when to say “I don’t know” is more useful across messy real-world domains than one that confabulates confidently, and it is notable that a lab whose stated mission is extending human will and judgment treats calibrated uncertainty as a first-class training target rather than a safety afterthought. The censorship-resistance training, validated on Cognition’s Propaganda and Censorship Eval, extends the same idea: trustworthiness as a capability you train, not a policy you bolt on.

    Finally, the open-weights safety tension is handled with unusual candor. Inkling posts the strongest adversarial FORTRESS score among the open models compared while keeping benign over-refusal low, and it was tested externally for CBRN, cyber, and loss-of-control capabilities. But everyone in this space knows fine-tuning can strip safety behavior from open weights, and TML ships a fine-tuning platform for this exact model. Their acknowledgment that they are actively studying how safety behavior survives fine-tuning on Tinker is the right thing to say, and it is also the open question that will define whether “safe open weights” is a coherent category at all.

    Key Takeaways

    • Inkling is Thinking Machines Lab’s first from-scratch, open-weights model: a Mixture-of-Experts transformer with 975B total parameters, 41B active, and a context window up to 1M tokens.
    • It was pretrained on 45 trillion tokens spanning text, images, audio, and video, and reasons natively over text, images, and audio without separate encoders.
    • A preview of Inkling-Small ships alongside it: a 276B-parameter MoE with just 12B active parameters that matches or beats its larger sibling on several benchmarks thanks to an improved pretraining recipe.
    • TML explicitly positions Inkling as a base for customization rather than the strongest overall model, leaning on multimodality, efficient thinking, and Tinker fine-tuning as the differentiators.
    • The launch demo shows Inkling fine-tuning itself: it wrote its own training objective and data, ran the job through the Tinker API, evaluated the result, and hot-swapped to its own new weights inside the OpenCode harness.
    • The self-fine-tuning target was a lipogram assistant that never uses the letter “e,” a behavior chosen precisely because prompting alone cannot reliably achieve it; the full loop completed in about 27 minutes.
    • Controllable thinking effort is a core feature: a setting swept from 0.2 to 0.99 traces a full performance-versus-tokens curve instead of a single benchmark point.
    • On Terminal Bench 2.1, Inkling matches Nemotron 3 Ultra’s score at roughly one third of the generated tokens, the release’s flagship efficiency claim.
    • Inkling was trained to run inside a variety of coding and agent harnesses, with tool sets and schemas randomized during training to reduce sensitivity to any particular harness.
    • On Design Arena’s blinded human-evaluated Agentic Web Dev leaderboard, Inkling scores 1257, among the strongest open-weights models and tied with Claude Opus 4.6.
    • Headline benchmark scores at effort 0.99 include SWEBench Verified 77.6%, SWEBench Pro Public 54.3%, Terminal Bench 2.1 63.8%, GPQA Diamond 87.2%, AIME 2026 97.1%, and HLE 29.7% text-only (46.0% with tools).
    • Agentic and general scores include MCP Atlas 74.1%, Tau 3 Banking 23.7%, and BrowseComp 77.1% with context management.
    • Vision results are strong for an open model: MMMU Pro 73.5%, CharXiv RQ 78.1%, rising to 82.0% when the model uses a Python tool for zooming and cropping during visual reasoning.
    • Audio results place it among the strongest open-weights audio models: VoiceBench 91.4%, MMAU 77.2%, and Audio MC 56.6%, well ahead of Qwen3-Omni and Nemotron Nano-Omni on the last.
    • The multimodal stack is encoder-free: audio enters as discrete dMel spectrograms and images as 40×40 pixel patches through a four-layer hMLP, both passed through a lightweight embedding layer and processed jointly with text tokens.
    • The MoE design largely follows DeepSeek-V3: 256 routed experts plus 2 shared experts per layer, 6 routed experts active per token, with a sigmoid router and auxiliary-loss-free load balancing.
    • Attention interleaves sliding-window and global layers at a 5:1 ratio with 8 KV heads, and uses a learned relative positional embedding instead of RoPE, which TML found extrapolates better to long sequences.
    • Short convolutions are applied after the key and value projections and on the attention and MLP residual branch outputs, an unusual architectural touch aimed at efficiency and long-context performance.
    • Training used a hybrid optimizer strategy, Muon for large matrix weights and Adam for everything else, with weight decay coupled to the square of the learning rate to keep weight magnitudes stable.
    • Post-training was bootstrapped with a small SFT phase on synthetic data generated by open-weights models including Kimi K2.5, with the large majority of compute spent on large-scale RL.
    • RL was scaled past 30 million rollouts across two long continuous runs, with reasoning performance on a held-out aggregate (AIME, HLE, GPQA, and others) improving log-linearly throughout.
    • Effort control was trained by varying the system message and per-token cost across rollouts, teaching the model to modulate its own thinking budget.
    • An emergent effect appeared during RL: the chain of thought compressed over training, dropping articles and connectives into a telegraphic style, driven purely by efficiency pressure rather than any targeted reward.
    • Inkling was TML’s first major training effort and ran on NVIDIA GB300 NVL72 systems; the lab says future models will push compute scale further across pretraining and RL.
    • Calibration was trained directly with RL against proper scoring rules on a large corpus of resolved real-world questions, treating well-placed confidence as a capability rather than a byproduct.
    • On ForecastBench without search, Inkling’s Brier Index of 61.1 beats GPT-5.5 (59.1) and Claude Opus 4.8 (54.6), and it stays competitive with search enabled and on Prophet Arena.
    • Instruction following was trained with two automated graders working together: a rubric grader scoring against a checklist and a claims grader that verifies each factual claim via agentic web search, improving helpfulness and reducing hallucination simultaneously.
    • Abstention-aware rewards on short-form factual QA taught the model to answer when confident and hedge or decline when not, with some prompts explicitly forcing or forbidding hedging so the user’s preference wins.
    • Inkling was trained to answer directly on topics subject to censorship, and Cognition’s Propaganda and Censorship Eval found strong censorship non-compliance.
    • On FORTRESS, Inkling posts the strongest adversarial refusal score (78.0%) of any compared open-weights model while keeping benign compliance high (95.9%), and scores 98.6% on StrongREJECT.
    • Safety testing covered CBRN, cyber, and loss-of-control capabilities plus human-AI threat vectors like sycophancy, vulnerable users, and manipulation, verified by commissioned external testers.
    • Inkling is available for fine-tuning on Tinker today with 64K and 256K context options at a 50% limited-time discount, plus a free Inkling Playground chat interface in the Tinker console.
    • Full weights are on Hugging Face, including an NVFP4 checkpoint for efficient inference on NVIDIA Blackwell, with API availability via Together, Fireworks, Modal, Databricks, and Baseten and inference support in SGLang, vLLM, TokenSpeed, and llama.cpp.
    • TML frames Inkling as the first in a family and as the intended background reasoning model for its previously announced real-time interaction models system.

    Detailed Summary

    What Inkling Is and Why It Exists

    Thinking Machines Lab frames its mission as building AI that extends human will and judgment, and Inkling as the logical next step after shipping the Tinker customization platform, previewing an interaction-focused AI system, and publishing research. Inkling is a Mixture-of-Experts transformer with 975B total and 41B active parameters, a context window up to 1M tokens, and pretraining on 45 trillion tokens of mixed text, image, audio, and video data. The lab is upfront that it is not the strongest model available. The pitch is breadth plus adaptability: a generalist trained across agentic, reasoning, coding, instruction-following, factuality, vision, and audio tasks rather than tuned to dominate one leaderboard, offered with full weights so people can make it their own. It launches with a preview sibling, Inkling-Small, at 276B total and 12B active parameters.

    The Self-Fine-Tuning Demo

    To demonstrate what customization means, TML asked Inkling to fine-tune itself. Running inside the OpenCode harness with access to Tinker, the model was told to become a lipogram assistant that never uses the letter “e.” Inkling drafted the plan, wrote an objective file with a scoring function (any response containing “e” scores zero), generated synthetic training data, launched a supervised fine-tuning run through the Tinker API, evaluated the checkpoint against its base self, and then staged a self-update so the supervisor relaunched the session on the new weights. The pipeline passed in about 27 minutes, and the updated model answered a test question about launching an LLM without a single “e.” It is a whimsical objective wrapped around a serious primitive: a model autonomously specifying, running, and adopting its own weight updates.

    Agentic Coding and Tool Use

    TML trained Inkling to operate inside many coding and agent harnesses, randomizing tool sets and schemas during training so the model does not overfit to one environment. The release showcases three demos: a one-shot job-application web app that then hosts an embedded browser-use agent operating its own interface; a nine-page, cohesively designed PDF food and travel journal produced from a single editorial prompt with web-verified details; and a server-authoritative multiplayer snake game refined over 40 iterations of feedback from GPT Codex acting as a reviewer. On benchmarks, Inkling posts 77.6% on SWEBench Verified, 54.3% on SWEBench Pro Public, and 63.8% on Terminal Bench 2.1, competitive within the open-weights field, and 1257 on Design Arena’s human-judged web dev leaderboard, in the same band as Claude Opus 4.6.

    Controllable Thinking Effort

    Rather than reporting a single operating point, TML sweeps Inkling’s effort setting from 0.2 to 0.99 and plots score against mean generated tokens on Terminal Bench 2.1, HLE, and IFBench, with competitors shown at their default settings. The headline result is efficiency: Inkling reaches Nemotron 3 Ultra’s Terminal Bench score at roughly a third of the tokens. The argument is that cost and latency are binding constraints in production, especially for interactive collaboration, so the full cost curve, not the peak score, is what developers should evaluate. Effort can be set from within the agent harness, and the ability was trained by varying system messages and per-token costs across RL rollouts.

    Native Multimodality Without Encoders

    Inkling is designed to serve as the background reasoning model for TML’s interaction models system, which requires real-time voice and vision collaboration. The multimodal components are trained from scratch with an encoder-free architecture: audio arrives as discrete dMel spectrograms and images as 40×40 pixel patches through a four-layer hMLP, both mapped through a lightweight embedding layer and processed jointly with text. The model transcribes speech, follows spoken instructions, reasons over long recordings, and answers questions about charts and diagrams, optionally using a Python tool to zoom and crop images mid-reasoning. Scores like 91.4% on VoiceBench and 82.0% on CharXiv RQ with Python place it among the strongest open-weights multimodal models, though still behind Gemini 3.1 Pro.

    Epistemics: Calibration, Forecasting, and Censorship Resistance

    TML groups calibration, instruction following, and censorship resistance under the banner of epistemics. Calibration was trained with RL against proper scoring rules on resolved real-world questions, and it shows: Inkling’s ForecastBench Brier Index of 61.1 without search beats GPT-5.5 and Claude Opus 4.8, and its Prophet Arena score sits close to the frontier. Instruction following used two complementary automated graders, a rubric checklist and a claims grader that verifies factual assertions through agentic web search, so recall-spraying to hack rubrics gets penalized by the factuality check. Targeted abstention-aware QA datasets taught the model to say “I don’t know” or give hedged best guesses when appropriate, while still complying when a user demands a forced guess. Finally, the model was trained to answer directly on censorship-prone topics, with Cognition’s Propaganda and Censorship Eval finding strong non-compliance with censorship patterns.

    Safety for an Open-Weights Release

    Inkling was trained to an internal behavioral spec across all modalities and then checked by commissioned external safety testers. Evaluations covered dangerous capabilities (CBRN, cyber, loss of control) and human-AI threat vectors including sycophancy, vulnerable users, and harmful manipulation. On FORTRESS, which pairs adversarial harmful requests with benign look-alikes, Inkling posts the strongest adversarial score among the compared open models (78.0%) without collapsing on the benign side (95.9%), and it scores 98.6% on StrongREJECT. TML acknowledges the open question hanging over every open-weights release: how safety behavior holds up under fine-tuning, which it says it is actively studying on Tinker.

    Architecture and Training Recipe

    The MoE layout follows DeepSeek-V3: 256 routed experts and 2 shared experts per layer with 6 routed experts active per token, a sigmoid-based router, and auxiliary-loss-free load balancing. Attention interleaves sliding-window and global layers 5:1 with 8 KV heads, and positions are encoded with a learned relative positional embedding that TML found outperforms and out-extrapolates RoPE. Short convolutions appear after the key and value projections and on residual branch outputs. Optimization was hybrid, Muon for large matrices and Adam elsewhere, with hyperparameter schedules drawn from the lab’s modular manifolds research and weight decay coupled to the square of the learning rate to keep weight norms stable. Post-training bootstrapped from a small SFT phase on synthetic data from open models including Kimi K2.5, then spent the bulk of compute on large-scale RL. Everything ran on NVIDIA GB300 NVL72 systems.

    RL at Scale and the Emergent Compression of Thought

    TML scaled asynchronous RL past 30 million rollouts across two long continuous runs, with performance on a held-out aggregate of reasoning evals improving log-linearly the whole way. Along the way an unplanned behavior emerged: the chain of thought became progressively more concise, shedding grammatical overhead into a telegraphic style (“We need to understand” becomes “We need determine”) while remaining comprehensible and leaving final answers unaffected. No reward targeted this; token efficiency pressure alone drove the compression, echoing an observation Cognition made while training SWE-1.7. It is a vivid example of optimization discovering its own shorthand.

    Inkling-Small

    The preview of Inkling-Small is arguably the sleeper story: with 12B active parameters against Inkling’s 41B, it matches or exceeds the larger model on a surprising number of benchmarks, including GPQA Diamond (88.3% vs 87.2%), IFBench (83.4% vs 79.8%), and CharXiv RQ with Python (83.4% vs 82.0%). TML attributes this to pretraining data and recipe improvements made after the big model trained, with both models sharing the same post-training stack. The clearest gaps favoring big Inkling are factuality (SimpleQA 43.9% vs 20.9%), Terminal Bench, and Tau 3 Banking. Full weights for Inkling-Small will be released once testing finishes, and its cost and latency profile targets high-volume workloads like coding, LLM grading, and synthetic data generation.

    Availability and the Ecosystem Play

    Inkling is on Tinker today with 64K and 256K context options at a limited-time 50% discount, plus a free Inkling Playground chat interface with integrated web search in the Tinker console so developers can get a feel for the model before committing to a run. The cookbook gained native Inkling support and three new audio recipes, and a new tml-renderer handles chat templates, tool calls, reasoning content, and multimodal inputs. Deployment partnerships span Together, Fireworks, Modal, Databricks, and Baseten for APIs; RadixArk for SGLang and Miles; Inferact for vLLM; Lightseek for TokenSpeed; Unsloth for llama.cpp; and Hugging Face for transformers integration. Full weights are on Hugging Face in both the original checkpoint and an NVFP4 checkpoint for NVIDIA Blackwell inference.

    Notable Quotes

    “Our mission is to build AI that extends human will and judgment.”

    Thinking Machines Lab, opening the Inkling announcement

    The company’s north star, and the lens through which the whole release (customization, calibration, open weights) is framed.

    “Inkling is not the strongest overall model available today, open or closed. Instead, a combination of qualities makes it a good open-weights base for customization: multimodal capabilities, efficient thinking, and availability on Tinker for fine-tuning.”

    Thinking Machines Lab, positioning the release

    A rare piece of launch-day honesty from a frontier lab, and the strategic thesis of the whole release.

    “Picking the right base model to fine-tune is a qualitative judgment that combines measurable benchmarks with the unique feel of a model that comes from playing with it.”

    Thinking Machines Lab, on why the Inkling Playground exists

    An argument that vibes are data, from the lab that built a playground into a fine-tuning console.

    “Cost and latency are often binding constraints in real-world applications, and low latency in particular is crucial for enabling collaboration and improvement through iteration.”

    Thinking Machines Lab, on controllable thinking effort

    The case for evaluating models on their full effort-versus-performance curve instead of a single benchmark point.

    “A model that’s confident in every answer it gives, including when it’s missing info and confabulates, forces the user to double-check everything.”

    Thinking Machines Lab, on why calibration was a training target

    The clearest one-line justification for treating calibrated uncertainty as a capability rather than a nicety.

    “Together, the two graders improve helpfulness and reduce hallucination at the same time, rather than trading one for the other.”

    Thinking Machines Lab, on pairing a rubric grader with a web-searching claims grader

    A neat solution to rubric hacking: verify every claim with agentic search so spraying plausible facts stops paying.

    “Safety is crucial for open-weights models. We’re continuing to study safety behavior and capability uplift in customizable models, including how safety behavior is impacted by fine-tuning on Tinker.”

    Thinking Machines Lab, on the open question of fine-tunable safety

    The acknowledgment that safety trained into open weights must survive the very customization the product sells.

    “Inkling is just the start: our first release in a model family we will continue to build on.”

    Thinking Machines Lab, on the roadmap

    Together with the GB300 compute note, a clear signal that larger and stronger family members are coming.

    Read the full announcement, including the interactive demos, effort curves, and complete benchmark tables, on the Thinking Machines Lab blog.

    Related Reading

  • SubQ 1.1 Small Explained: How Subquadratic Sparse Attention Hits 98% Retrieval at 12 Million Tokens With 64.5x Less Compute Than Dense Attention

    Subquadratic, a frontier AI research and infrastructure company, has released the model card and technical report for SubQ 1.1 Small, a long-context language model built on a new attention mechanism the company calls Subquadratic Sparse Attention (SSA). The headline claim is unusual in two directions at once: the model retains 98% single-fact retrieval accuracy at 12 million tokens, roughly twelve times the length it was primarily trained on, while cutting attention compute by 64.5x against dense attention at a 1 million token context. The deeper argument in the report is not really about a single model at all. It is about what happens to the entire retrieval-and-orchestration stack once reasoning over a complete artifact stops being prohibitively expensive.

    TLDR

    SubQ 1.1 Small is a small long-context model that replaces the dense attention of an existing open-weight frontier model with Subquadratic Sparse Attention, a learned, content-dependent sparse attention mechanism that scales linearly in compute and memory rather than quadratically. On retrieval it posts 99.12% on NVIDIA’s 13-task RULER suite at 128K tokens and 100% needle-in-a-haystack accuracy at 1M and 2M tokens, holding at 98% out to 6M and 12M tokens while attending to only 0.13% of token pairs. It keeps competitive general ability, scoring 85.4% on GPQA Diamond and 89.7% pass@4 on LiveCodeBench v6, and reaches 13% on the long-horizon AutomationBench Finance agentic benchmark, close to Opus 4.8 and GPT-5.5 and well ahead of mid and small tiers. The efficiency story is a scaling win rather than a constant-factor one: 64.5x fewer attention FLOPs than dense attention at 1M tokens and 56x faster than FlashAttention-2 on a single attention layer. The report frames cheap long-context compute as a research accelerator that let the team run more than one hundred million-token experiments and find a training recipe (long-context continued pretraining is the strongest lever) rather than guess at one, positions SSA against FlashAttention, DeepSeek’s Lightning Indexer line, state space models like Mamba, and hybrids, invokes Sutton’s Bitter Lesson to argue that RAG, chunking, and agentic scaffolding are partly workarounds for context scarcity, and was independently verified by Appen. Deployment is starting with design partners now, with a 2M to 12M token lineup planned by year end.

    Thoughts

    The most interesting move in this report is the framing, not the benchmark. Subquadratic plants its flag on Richard Sutton’s Bitter Lesson and argues that much of the modern AI stack, the retrieval pipelines, the chunkers, the re-rankers, the agentic orchestration, is scaffolding built around a single computational constraint: dense attention costs grow with the square of context length. If that constraint relaxes, a lot of hand-engineered machinery that exists to feed a model the right fragments at the right moment starts to look like the task-specific pipelines that learned representations eventually displaced. That is a genuinely provocative thesis, and it is the right lens for reading the rest of the document. The company is not selling a longer context window as a feature. It is betting that whole-artifact reasoning is a different shape of capability than retrieval over fragments, and that fragmentation destroys the cross-references a contract or a codebase actually depends on before the model ever sees them.

    The part of the paper most teams will undervalue is the claim that the real payoff of efficient attention is not cheaper inference but cheaper experimentation. A dense long-context training campaign is expensive enough that most groups get a handful of attempts and are forced to guess at the recipe. Subquadratic says SSA let them run more than a hundred experiments across six model generations with per-step iteration under a minute at million-token context, which is how they discovered that long-context continued pretraining, not clever post-training, was the dominant lever. If that holds, algorithmic efficiency becomes a first-class scaling variable alongside parameters and data, because capability becomes responsive to iteration velocity rather than raw compute alone. It reframes efficiency from a deployment line item into a research multiplier, and that is a more durable advantage than any single benchmark number.

    The generalization result deserves scrutiny precisely because it is so clean. A model trained overwhelmingly at 1M tokens, with a sliver at 2M and nothing beyond, holds 98% retrieval at 12M. The proposed explanation is that SSA routes attention by content relevance rather than fixed positional pattern, so there may simply be no obvious length boundary once the routing behavior is learned. That is plausible and the report is careful to say the 12M result emerged rather than being designed for. But single-needle NIAH is a deliberately clean probe with one target and a binary answer. The far harder RULER suite is only reported at 128K, the longest standardized length in the original benchmark, so the multi-hop, aggregation, and distractor-heavy capability that whole-artifact reasoning actually requires has public numbers at 128K, not at 12M. The honest read is that precise retrieval generalizes spectacularly and composite reasoning at extreme length is still an open question the report does not over-claim on.

    What lends the report credibility is how much counter-evidence it volunteers. It walks through MiniMax abandoning its hybrid M1 architecture and returning to full attention for M2 after efficient variants showed multi-hop reasoning deficits at scale. It admits that earlier SubQ checkpoints improved retrieval while regressing on knowledge benchmarks, forcing dedicated capability-balancing work. It describes catching a case where the MRCR benchmark moved up while the model felt worse in real workflow spot-checks, and switching its development signal to RULER as a result. That last point is a quietly important methodological argument: benchmark score and deployment behavior diverged enough to change checkpoint selection, which is a warning every team shipping long-context models should internalize. A vendor confident enough to show where its own metrics misled it is more trustworthy than one that only shows the wins.

    A few caveats keep the enthusiasm grounded. AutomationBench Finance at 13% is genuinely strong relative to peers, but it is a low absolute score across the board, including for GPT-5.5 at 18% and Opus 4.8 at 16%, so this is early evidence of agentic transfer rather than proof of a finished agent. The efficiency comparisons isolate a single attention layer rather than full end-to-end model throughput, which is the right way to expose the scaling shape but not the same as a wall-clock serving benchmark. The model is built from an unnamed donor open-weight frontier model, so some of its general-knowledge and coding strength is inherited rather than created here. And the most aggressive claims about the future, a 2M to 12M lineup and much higher sparsity, are roadmap, not released artifacts. None of that undercuts the core result. It just means the right posture is to treat SubQ 1.1 Small as a strong proof of concept for an architecture that, if it scales as advertised, could quietly remove a layer of the AI stack that everyone currently takes for granted.

    Key Takeaways

    • SubQ 1.1 Small is a long-context language model from Subquadratic AI, built on a new attention mechanism called Subquadratic Sparse Attention (SSA), released June 16, 2026 alongside a model card and technical report.
    • SSA is a learned, content-dependent sparse attention mechanism that scales linearly in both compute and memory with sequence length, rather than quadratically like dense attention.
    • The central result is context-length generalization: the model was trained primarily at 1M tokens, with some training at 2M and none beyond, yet retrieval held far past the training window.
    • Needle-in-a-haystack accuracy is 100% at 1M and 2M tokens and 98% at both 6M and 12M tokens, roughly twelve times the primary training length.
    • At 12M tokens the model attends to only 0.13% of token pairs, close to a 1,000x reduction in attention relationships, while still retrieving accurately.
    • On NVIDIA’s 13-task RULER benchmark at 128K tokens, SubQ 1.1 Small scores 99.12%, with the remaining errors concentrated in aggregation-style tasks rather than retrieval.
    • RULER tests beyond single-fact lookup: single-key and multi-key retrieval, common-word and frequent-word extraction, and multi-hop variable tracing across positions.
    • At 1M tokens, SSA requires 64.5x fewer attention FLOPs than dense attention (3.9 PFLOP versus 252 PFLOP per attention layer).
    • On a single attention layer, SSA runs 56x faster than FlashAttention-2 at 1M tokens (966 ms versus 54,164 ms on an H100), reaching parity near 16K tokens and pulling away as context grows.
    • The efficiency gain is a scaling-law win, not a constant-factor speedup: the advantage over dense attention grows as context length increases.
    • On general knowledge, SubQ 1.1 Small scores 85.4% on GPQA Diamond (pass@1), below GPT-5.5 (93.2) and Opus 4.8 (92), near Sonnet 4.6 and GPT-5.4-mini (87.5), and above GPT-5.4-nano (81.7) and Haiku 4.5 (67.2).
    • On coding, it reaches 89.7% pass@4 on LiveCodeBench v6, close to the absolute frontier (GPT-5.5 92, Opus 4.8 92.2) and ahead of the smaller tiers.
    • On AutomationBench Finance, a long-horizon agentic benchmark, it scores 13%, close to Opus 4.8 (16%) and GPT-5.5 (18%) and ahead of Sonnet 4.6 (8%), Haiku 4.5 (3%), and GPT-5.4-mini (0%). Absolute scores are low across all models.
    • The model was not trained from scratch. The team converted an existing open-weight frontier model by replacing dense attention with SSA, then built long-context ability through staged context extension and continued pretraining.
    • Context was extended in stages (262K, 512K, 1M, 2M) using YaRN positional scaling, with long-context continued pretraining performed between extension stages on naturally long data: books, long documents, and repository-scale code.
    • Roughly one trillion tokens of continued pretraining were performed, most of it at the 1M-token stage.
    • Long-context continued pretraining was the most consistent predictor of long-context retrieval gains across the experiments, more so than post-training tweaks.
    • The team ran more than one hundred long-context experiments across six major model generations, which the report argues is only possible because SSA made million-token iteration cheap (under a minute per step).
    • Capability balance was a recurring challenge: gains in long-context retrieval often regressed short-context knowledge and reasoning unless training was explicitly managed for both.
    • Benchmark scores and real deployment behavior diverged. The MRCR benchmark moved up while qualitative workflow spot-checks got worse, so the team switched its primary development signal to RULER.
    • The report frames RAG, chunking, summarization, and agentic orchestration as scaffolding built around context scarcity, drawing an analogy to Sutton’s Bitter Lesson, where hand-engineered mechanisms get displaced by larger-scale learning.
    • SSA is positioned against FlashAttention (a memory optimization that does not change quadratic compute), fixed-pattern sparse attention, DeepSeek’s learned sparse line, state space models, and hybrid architectures.
    • DeepSeek’s Lightning Indexer (used in DSA and CSA) is the closest published comparison. Its quadratic scoring overtakes the sparse attention it feeds around 52,000 tokens, reaching roughly 16x the attention cost at 1M and 190x at 12M.
    • State space models like Mamba achieve linear cost through a compressed fixed-size state, but that compression is lossy and weakens exact retrieval, which is why production efficient models are usually hybrids with some dense attention layers retained.
    • MiniMax is cited as a cautionary case: it moved from a hybrid M1 to a full-attention M2 after hybrids showed multi-hop reasoning deficits at scale and less mature supporting infrastructure.
    • The benchmark results were independently verified by Appen, a third-party evaluation firm.
    • The named use cases are financial analysis and due diligence, legal and contract work, and software engineering (architecture-level reasoning, cross-file refactoring, dependency tracing, planning, review, and long-horizon memory).
    • Sparsity settings were deliberately conservative, tuned for maximum context length rather than maximum sparsity. Limited experiments at 4x the sparsity reported positive early results.
    • The training infrastructure used a memory-scaling ladder: single node, intra-node sequence parallelism, CPU offload, multi-node sequence parallelism, nested offloading, and Ring Attention for the longest contexts.
    • Beyond about 8M tokens, BF16 numerical underflow and stability became practical constraints on evaluation.
    • The technical report is authored by Saul Ramirez, Alex Whedon, Ashmal Vayani, and Phong Vo of Subquadratic AI.
    • Deployment is starting with a first cohort of design partners, with broader rollout through the quarter and a general model lineup ranging from 2M to 12M tokens by the end of the year.
    • The company’s framing line is “Efficiency is intelligence,” and its broader thesis is that the point is not bigger context windows for their own sake but reasoning directly over complete artifacts with less surrounding scaffolding.

    Detailed Summary

    The problem: whole-artifact reasoning and context scarcity

    The report opens by naming a class of tasks it calls whole-artifact reasoning: problems whose structure requires reasoning across a complete artifact rather than over isolated fragments. A legal agreement may define a term on page 2, qualify it on page 12, carve out an exception on page 46, and amend it in a schedule. A function may be defined in one file, called from forty others, and constrained by invariants encoded in the architecture rather than in comments. A financial review may require connecting filings, earnings reports, contracts, and internal records. In each case the difficulty is not locating a passage, it is reasoning over relationships distributed throughout a large artifact. Most production systems do not do this directly. They rely on retrieval pipelines, chunking, summaries, and agentic workflows that partition information and reconstruct fragments at inference time, because dense attention scales quadratically with context length and makes direct reasoning over large artifacts expensive. Subquadratic argues that much of the modern AI stack is therefore designed to manage context scarcity rather than reason over complete artifacts, and it connects this to Sutton’s Bitter Lesson: sophisticated hand-engineered mechanisms historically get displaced once larger-scale learning becomes practical.

    What SSA is and the three requirements it targets

    Subquadratic Sparse Attention is a content-dependent sparse attention mechanism designed to satisfy three requirements at once, a combination the report argues prior approaches never achieved in a practical long-context system. First, dense-attention-level retrieval and reasoning quality, which requires routing that is content-dependent (determined by the tokens themselves) rather than driven by a fixed positional pattern. Second, subquadratic scaling, where selection, retrieval, and attention are each linear in sequence length so the mechanism is linear end to end, not only within the attention read. Third, full-context training with standard autoregressive generation, so the model can optimize over the entire context during training while keeping efficient token-by-token decoding at inference. The internal mechanism by which SSA achieves this is held back as outside the scope of the report, which focuses instead on the requirements and the experimental program that followed.

    Where SSA sits among prior approaches

    The background section is effectively a taxonomy of long-context modeling. FlashAttention is treated not as a competitor but as the standard dense-attention baseline: it solved the memory problem by never materializing the full attention matrix, but it left the quadratic compute cost untouched, so doubling context still quadruples attention computation. Fixed-pattern sparse attention (sliding-window, strided, as in Longformer, BigBird, and the sliding window in Gemma) scales well but sacrifices content-dependent routing and tends to fail on retrieval benchmarks like RULER. Compression methods like Multi-head Latent Attention reduce KV-cache memory at inference but do not change the quadratic prefill cost. Learned sparse attention, exemplified by DeepSeek’s Native Sparse Attention and its Lightning Indexer, learns where to route but pays a quadratic cost in the indexer itself. State space models and linear attention (Mamba, Mamba-2 and Mamba-3, RetNet, RWKV, gated delta networks) achieve linear cost through a compressed fixed-size state, but that compression is lossy and weak on exact retrieval. Hybrids (Jamba, Kimi Linear, Qwen3 Next, Nemotron) keep a few dense layers to preserve retrieval, which means the quadratic component still dominates at long context. System-level workarounds (RAG, agentic frameworks, recursive language models) move retrieval outside the model entirely. The report’s stated open problem is to combine subquadratic scaling end to end with content-dependent retrieval, arbitrary-position access, and practical ultra-long-context training in one system, which it claims no widely deployed architecture provides and which SSA targets.

    Training: conversion, staged context extension, and continued pretraining

    Rather than training from scratch, the team converted an existing open-weight frontier model that supported a 262K-token context by replacing its dense attention with SSA. They then extended the context window in stages (262K to 512K to 1M to 2M) using YaRN to rescale positional representations, performing long-context continued pretraining between extension stages rather than jumping straight to the final length. The training mixture emphasized naturally long data such as books, long documents, and repository-scale code, packed to the target length with document separators and without masking cross-document attention boundaries. Most continued-pretraining tokens were trained at the 1M-token stage, with roughly one trillion tokens total. Post-training played a separate role: shaping how the long-context capability was expressed while preserving reasoning, coding, and instruction following. The team explored sample-level loss aggregation to keep a few extremely long examples from dominating gradient updates, and staged the post-training corpus across synthetic retrieval tasks, long-context reasoning, coding, educational material, and general instruction following, alternating capability-building phases with recovery phases.

    Results: retrieval, knowledge, coding, and agentic tasks

    On retrieval, SubQ 1.1 Small scores 99.12% on the 13-task RULER average at 128K, with errors concentrated in aggregation-style tasks like common-word and frequent-word extraction. On needle-in-a-haystack, evaluated on 50 held-out UUID samples per length, it scores 100% at 1M and 2M (within the training window) and 98% at 6M and 12M (held out), attending to only 0.13% of token pairs at 12M. On knowledge, GPQA Diamond pass@1 is 85.4%, landing between the small and mid frontier tiers and confirming that long-context optimization need not sacrifice reasoning, a result the report credits to its capability-balancing stages after earlier checkpoints showed retrieval gains coming at the cost of knowledge. On coding, LiveCodeBench v6 pass@4 is 89.7%, and the report notes coding data played a dual role, also improving non-code long-context retrieval because code is dense with the cross-position dependencies that train general routing. On long-horizon agentic work, AutomationBench Finance is 13%, where agents must discover the right endpoints among roughly 500 across 47 applications, make interdependent API calls, follow layered business rules, and ignore seeded distractors, graded on binary end-state correctness with no partial credit.

    Efficiency and the DeepSeek comparison

    Efficiency is measured on one attention layer against a dense baseline on the same backbone. Per-forward-pass attention FLOPs scale from a 2.1x reduction at 32K to 8x at 128K, 31.5x at 512K, and 64.5x at 1M tokens (3.9 PFLOP for SSA versus 252 PFLOP for dense). Measured against FlashAttention-2 in isolation, SSA reaches parity near 16K tokens and pulls away to 56x at 1M, where it runs in 966 ms versus 54,164 ms on an H100. The report devotes a discussion section to DeepSeek’s sparse attention line as the closest published comparison. DeepSeek’s Lightning Indexer is a learned selector, but it is a full-attention distilled transformer, so it scales quadratically: in a V3.2-style configuration the indexer is cheaper than the sparse attention it feeds only below about 52,000 tokens, then overtakes it, reaching roughly 16x the attention cost at 1M tokens and 190x at 12M. SSA targets that same selection role with a selector the report says is dramatically cheaper and linear throughout, and notes SSA could conceptually replace the selector over either uncompressed or compressed representations.

    Efficiency as a research accelerator and the evaluation lessons

    A recurring theme is that the most valuable effect of cheap long-context compute was on the research loop, not just inference. Where a dense campaign would allow a handful of attempts, SSA enabled more than a hundred experiments across six model generations with per-step iteration under a minute at million-token context. That throughput is what surfaced the finding that long-context continued pretraining is the strongest lever, and it leads the authors to argue that algorithmic efficiency should be treated as a first-class scaling variable alongside model and dataset size. The report is unusually candid about evaluation pitfalls. It describes how the MRCR benchmark diverged from deployment behavior, with MRCR-optimized checkpoints often feeling worse on repository-scale code reasoning, multi-document synthesis, and contract analysis, which pushed the team to rely on RULER and a fixed set of qualitative workflow spot-checks as development signals. It also cites MiniMax returning from a hybrid M1 to a full-attention M2 as evidence that reducing asymptotic cost is not sufficient on its own if retrieval quality, reasoning at scale, and system maturity are not preserved at the same time.

    Implications, availability, and what comes next

    The report’s deployment argument is that the most important enterprise implication of long-context models is not larger windows but the ability to reason directly over complete or more-complete artifacts, moving retrieval, re-ranking, and orchestration logic into the model where the task is naturally whole-artifact rather than naturally decomposable. It is careful not to declare retrieval obsolete: for corpora larger than any plausible context window, fast-changing knowledge, and genuinely multi-stage workflows, RAG and orchestration remain the right tools. The narrower claim is that the class of scaffolding that exists only to compensate for context limits gets smaller as efficient long-context models extend the reachable window. The benchmark results were independently verified by Appen. Subquadratic is deploying SubQ 1.1 Small with a first cohort of design partners now, with broader rollout through the quarter and a general lineup spanning 2M to 12M tokens planned by the end of the year, and it flags much higher sparsity as future work.

    Notable Quotes

    “Much of the modern AI stack is therefore designed to manage context scarcity rather than reason over complete artifacts directly.”

    SubQ-1.1-Small Technical Report, framing retrieval and orchestration as workarounds for an architectural limit

    “The hybrid has moved the line, but not changed its shape.”

    SubQ-1.1-Small Technical Report, on why hybrid models keep their quadratic component at long context

    “A routing mechanism intended to make long context affordable becomes the dominant long-context cost, reintroducing quadratic scaling after providing scalar compute savings.”

    SubQ-1.1-Small Technical Report, on DeepSeek’s Lightning Indexer overtaking the attention it feeds

    “If the cost of long-context experiments is too high, teams are forced to guess at the recipe. If the cost falls far enough, they can search for it.”

    SubQ-1.1-Small Technical Report, on efficient attention as a research accelerator

    “Fragmentation systematically destroys those relationships before the model ever sees them.”

    SubQ-1.1-Small Technical Report, on why chunking hurts whole-artifact reasoning

    “Holding the whole artifact in context changes the shape of the task rather than only the speed of it.”

    SubQ-1.1-Small Technical Report, on the difference between bigger windows and direct reasoning

    “The value of SSA is therefore not only that it makes long-context inference cheaper. It makes long-context experimentation cheaper.”

    SubQ-1.1-Small Technical Report, conclusion

    Read the full SubQ 1.1 Small technical report and model card here.

    Related Reading

    • Subquadratic (subq.ai) the company behind SubQ 1.1 Small and the Subquadratic Sparse Attention architecture, where you can join the waitlist.
    • The Bitter Lesson by Richard Sutton the short essay whose argument the report leans on, that hand-engineered mechanisms lose to general methods that scale with computation.
    • Attention Is All You Need the original Transformer paper that introduced the dense attention whose quadratic cost SSA is built to remove.
    • RULER (arXiv) NVIDIA’s long-context benchmark that the report uses as its primary retrieval signal, and that fixed-pattern sparse methods historically struggle with.
    • Retrieval-augmented generation (Wikipedia) background on the RAG approach that the report frames as scaffolding around context scarcity rather than a permanent fixture.