PJFP.com

Pursuit of Joy, Fulfillment, and Purpose

Tag: Mixture of Experts

  • Inkling: Thinking Machines Lab Releases Its First Open-Weights Model, a 975B Multimodal Mixture-of-Experts With Controllable Thinking Effort That Can Fine-Tune Itself on Tinker

    Thinking Machines Lab, the AI startup founded by former OpenAI CTO Mira Murati, has released Inkling, its first open-weights model trained from scratch. Inkling is a 975 billion parameter Mixture-of-Experts transformer (41B active) with a context window of up to 1 million tokens, native multimodal reasoning over text, images, and audio, and a dial for controllable thinking effort. The lab is explicit that Inkling is not the strongest model in the world. It is pitched as something arguably more useful: a broad, balanced, customizable foundation you can fine-tune on Tinker, with the full weights on Hugging Face. The announcement even includes a demo where Inkling fine-tunes itself and swaps in its own new weights.

    TLDR

    Thinking Machines Lab released Inkling, a 975B-total, 41B-active Mixture-of-Experts model pretrained on 45 trillion tokens of text, images, audio, and video, alongside a preview of Inkling-Small (276B total, 12B active). The release covers the model’s generalist benchmark profile across reasoning, agentic coding, tool use, vision, and audio; a controllable thinking effort setting that lets developers trade performance against tokens (matching Nemotron 3 Ultra on Terminal Bench 2.1 at roughly a third of the tokens); an encoder-free multimodal architecture using dMel spectrograms and hMLP image patches; a training recipe combining Muon and Adam with weight decay coupled to the learning rate; RL scaled past 30 million rollouts with log-linearly improving reasoning and an emergent compression of the chain of thought; an epistemics push covering calibration, forecasting (where it beats several frontier models), abstention, and censorship resistance; the strongest FORTRESS adversarial safety score among compared open-weights models; a headline-grabbing demo of the model fine-tuning itself into a lipogram assistant via Tinker; and day-one availability on Tinker (at a 50% discount), Hugging Face, and inference partners including Together, Fireworks, Modal, Databricks, Baseten, vLLM, SGLang, and llama.cpp.

    Thoughts

    The most striking thing about this launch is its honesty. Nearly every frontier release leads with a claim to be the best at something, and the fine print walks it back. Thinking Machines Lab says plainly that Inkling is not the strongest model available, open or closed, and then makes the case that “strongest” is the wrong axis for most real buyers. If you are going to run a model millions of times inside a product, what you care about is the cost curve, the adaptability, and whether you can shape it to your workflow. That framing conveniently matches their business (Tinker sells fine-tuning), but it also matches how production AI actually gets deployed, where cost and latency are binding constraints and a benchmark crown is trivia.

    The self-fine-tuning demo deserves more attention than it will probably get. Asked to become a lipogram assistant that never uses the letter “e” (a behavior prompting alone cannot reliably produce), Inkling wrote its own training objective and scoring function, generated its own synthetic data, launched the run on Tinker, evaluated the result against its base self, and then staged a weight swap so the improved checkpoint took over the session. That is a closed loop of specify, train, evaluate, and self-update, packaged as a cute product demo. The loop is the primitive behind every serious conversation about recursive self-improvement, and here it is running as a marketing asset with a 27 minute wall clock. The gap between “toy objective” and “economically meaningful objective” is now a question of reward design, not plumbing.

    Controllable thinking effort is the feature I expect developers to care about most. Instead of publishing a single score, TML publishes a curve: sweep the effort setting from 0.2 to 0.99 and watch performance trade against generated tokens. Inkling reportedly matches Nemotron 3 Ultra on Terminal Bench 2.1 while spending about a third of the tokens. Benchmarks reported as single points hide exactly this, and a model that reaches a target score cheaply beats a model that scores two points higher at triple the cost in any high-volume workload. Expect effort curves to become standard marketing for open models, the way context length became standard a couple of years ago.

    The epistemics section is quietly the most differentiated part of the release. TML trained calibration directly, running RL against proper scoring rules on resolved real-world questions, and pairing a rubric grader with a claims grader that does agentic web search to verify each factual assertion. The result is a model that beats GPT-5.5 and Claude Opus 4.8 on ForecastBench without search and holds its own on Prophet Arena. A model that knows when to say “I don’t know” is more useful across messy real-world domains than one that confabulates confidently, and it is notable that a lab whose stated mission is extending human will and judgment treats calibrated uncertainty as a first-class training target rather than a safety afterthought. The censorship-resistance training, validated on Cognition’s Propaganda and Censorship Eval, extends the same idea: trustworthiness as a capability you train, not a policy you bolt on.

    Finally, the open-weights safety tension is handled with unusual candor. Inkling posts the strongest adversarial FORTRESS score among the open models compared while keeping benign over-refusal low, and it was tested externally for CBRN, cyber, and loss-of-control capabilities. But everyone in this space knows fine-tuning can strip safety behavior from open weights, and TML ships a fine-tuning platform for this exact model. Their acknowledgment that they are actively studying how safety behavior survives fine-tuning on Tinker is the right thing to say, and it is also the open question that will define whether “safe open weights” is a coherent category at all.

    Key Takeaways

    • Inkling is Thinking Machines Lab’s first from-scratch, open-weights model: a Mixture-of-Experts transformer with 975B total parameters, 41B active, and a context window up to 1M tokens.
    • It was pretrained on 45 trillion tokens spanning text, images, audio, and video, and reasons natively over text, images, and audio without separate encoders.
    • A preview of Inkling-Small ships alongside it: a 276B-parameter MoE with just 12B active parameters that matches or beats its larger sibling on several benchmarks thanks to an improved pretraining recipe.
    • TML explicitly positions Inkling as a base for customization rather than the strongest overall model, leaning on multimodality, efficient thinking, and Tinker fine-tuning as the differentiators.
    • The launch demo shows Inkling fine-tuning itself: it wrote its own training objective and data, ran the job through the Tinker API, evaluated the result, and hot-swapped to its own new weights inside the OpenCode harness.
    • The self-fine-tuning target was a lipogram assistant that never uses the letter “e,” a behavior chosen precisely because prompting alone cannot reliably achieve it; the full loop completed in about 27 minutes.
    • Controllable thinking effort is a core feature: a setting swept from 0.2 to 0.99 traces a full performance-versus-tokens curve instead of a single benchmark point.
    • On Terminal Bench 2.1, Inkling matches Nemotron 3 Ultra’s score at roughly one third of the generated tokens, the release’s flagship efficiency claim.
    • Inkling was trained to run inside a variety of coding and agent harnesses, with tool sets and schemas randomized during training to reduce sensitivity to any particular harness.
    • On Design Arena’s blinded human-evaluated Agentic Web Dev leaderboard, Inkling scores 1257, among the strongest open-weights models and tied with Claude Opus 4.6.
    • Headline benchmark scores at effort 0.99 include SWEBench Verified 77.6%, SWEBench Pro Public 54.3%, Terminal Bench 2.1 63.8%, GPQA Diamond 87.2%, AIME 2026 97.1%, and HLE 29.7% text-only (46.0% with tools).
    • Agentic and general scores include MCP Atlas 74.1%, Tau 3 Banking 23.7%, and BrowseComp 77.1% with context management.
    • Vision results are strong for an open model: MMMU Pro 73.5%, CharXiv RQ 78.1%, rising to 82.0% when the model uses a Python tool for zooming and cropping during visual reasoning.
    • Audio results place it among the strongest open-weights audio models: VoiceBench 91.4%, MMAU 77.2%, and Audio MC 56.6%, well ahead of Qwen3-Omni and Nemotron Nano-Omni on the last.
    • The multimodal stack is encoder-free: audio enters as discrete dMel spectrograms and images as 40×40 pixel patches through a four-layer hMLP, both passed through a lightweight embedding layer and processed jointly with text tokens.
    • The MoE design largely follows DeepSeek-V3: 256 routed experts plus 2 shared experts per layer, 6 routed experts active per token, with a sigmoid router and auxiliary-loss-free load balancing.
    • Attention interleaves sliding-window and global layers at a 5:1 ratio with 8 KV heads, and uses a learned relative positional embedding instead of RoPE, which TML found extrapolates better to long sequences.
    • Short convolutions are applied after the key and value projections and on the attention and MLP residual branch outputs, an unusual architectural touch aimed at efficiency and long-context performance.
    • Training used a hybrid optimizer strategy, Muon for large matrix weights and Adam for everything else, with weight decay coupled to the square of the learning rate to keep weight magnitudes stable.
    • Post-training was bootstrapped with a small SFT phase on synthetic data generated by open-weights models including Kimi K2.5, with the large majority of compute spent on large-scale RL.
    • RL was scaled past 30 million rollouts across two long continuous runs, with reasoning performance on a held-out aggregate (AIME, HLE, GPQA, and others) improving log-linearly throughout.
    • Effort control was trained by varying the system message and per-token cost across rollouts, teaching the model to modulate its own thinking budget.
    • An emergent effect appeared during RL: the chain of thought compressed over training, dropping articles and connectives into a telegraphic style, driven purely by efficiency pressure rather than any targeted reward.
    • Inkling was TML’s first major training effort and ran on NVIDIA GB300 NVL72 systems; the lab says future models will push compute scale further across pretraining and RL.
    • Calibration was trained directly with RL against proper scoring rules on a large corpus of resolved real-world questions, treating well-placed confidence as a capability rather than a byproduct.
    • On ForecastBench without search, Inkling’s Brier Index of 61.1 beats GPT-5.5 (59.1) and Claude Opus 4.8 (54.6), and it stays competitive with search enabled and on Prophet Arena.
    • Instruction following was trained with two automated graders working together: a rubric grader scoring against a checklist and a claims grader that verifies each factual claim via agentic web search, improving helpfulness and reducing hallucination simultaneously.
    • Abstention-aware rewards on short-form factual QA taught the model to answer when confident and hedge or decline when not, with some prompts explicitly forcing or forbidding hedging so the user’s preference wins.
    • Inkling was trained to answer directly on topics subject to censorship, and Cognition’s Propaganda and Censorship Eval found strong censorship non-compliance.
    • On FORTRESS, Inkling posts the strongest adversarial refusal score (78.0%) of any compared open-weights model while keeping benign compliance high (95.9%), and scores 98.6% on StrongREJECT.
    • Safety testing covered CBRN, cyber, and loss-of-control capabilities plus human-AI threat vectors like sycophancy, vulnerable users, and manipulation, verified by commissioned external testers.
    • Inkling is available for fine-tuning on Tinker today with 64K and 256K context options at a 50% limited-time discount, plus a free Inkling Playground chat interface in the Tinker console.
    • Full weights are on Hugging Face, including an NVFP4 checkpoint for efficient inference on NVIDIA Blackwell, with API availability via Together, Fireworks, Modal, Databricks, and Baseten and inference support in SGLang, vLLM, TokenSpeed, and llama.cpp.
    • TML frames Inkling as the first in a family and as the intended background reasoning model for its previously announced real-time interaction models system.

    Detailed Summary

    What Inkling Is and Why It Exists

    Thinking Machines Lab frames its mission as building AI that extends human will and judgment, and Inkling as the logical next step after shipping the Tinker customization platform, previewing an interaction-focused AI system, and publishing research. Inkling is a Mixture-of-Experts transformer with 975B total and 41B active parameters, a context window up to 1M tokens, and pretraining on 45 trillion tokens of mixed text, image, audio, and video data. The lab is upfront that it is not the strongest model available. The pitch is breadth plus adaptability: a generalist trained across agentic, reasoning, coding, instruction-following, factuality, vision, and audio tasks rather than tuned to dominate one leaderboard, offered with full weights so people can make it their own. It launches with a preview sibling, Inkling-Small, at 276B total and 12B active parameters.

    The Self-Fine-Tuning Demo

    To demonstrate what customization means, TML asked Inkling to fine-tune itself. Running inside the OpenCode harness with access to Tinker, the model was told to become a lipogram assistant that never uses the letter “e.” Inkling drafted the plan, wrote an objective file with a scoring function (any response containing “e” scores zero), generated synthetic training data, launched a supervised fine-tuning run through the Tinker API, evaluated the checkpoint against its base self, and then staged a self-update so the supervisor relaunched the session on the new weights. The pipeline passed in about 27 minutes, and the updated model answered a test question about launching an LLM without a single “e.” It is a whimsical objective wrapped around a serious primitive: a model autonomously specifying, running, and adopting its own weight updates.

    Agentic Coding and Tool Use

    TML trained Inkling to operate inside many coding and agent harnesses, randomizing tool sets and schemas during training so the model does not overfit to one environment. The release showcases three demos: a one-shot job-application web app that then hosts an embedded browser-use agent operating its own interface; a nine-page, cohesively designed PDF food and travel journal produced from a single editorial prompt with web-verified details; and a server-authoritative multiplayer snake game refined over 40 iterations of feedback from GPT Codex acting as a reviewer. On benchmarks, Inkling posts 77.6% on SWEBench Verified, 54.3% on SWEBench Pro Public, and 63.8% on Terminal Bench 2.1, competitive within the open-weights field, and 1257 on Design Arena’s human-judged web dev leaderboard, in the same band as Claude Opus 4.6.

    Controllable Thinking Effort

    Rather than reporting a single operating point, TML sweeps Inkling’s effort setting from 0.2 to 0.99 and plots score against mean generated tokens on Terminal Bench 2.1, HLE, and IFBench, with competitors shown at their default settings. The headline result is efficiency: Inkling reaches Nemotron 3 Ultra’s Terminal Bench score at roughly a third of the tokens. The argument is that cost and latency are binding constraints in production, especially for interactive collaboration, so the full cost curve, not the peak score, is what developers should evaluate. Effort can be set from within the agent harness, and the ability was trained by varying system messages and per-token costs across RL rollouts.

    Native Multimodality Without Encoders

    Inkling is designed to serve as the background reasoning model for TML’s interaction models system, which requires real-time voice and vision collaboration. The multimodal components are trained from scratch with an encoder-free architecture: audio arrives as discrete dMel spectrograms and images as 40×40 pixel patches through a four-layer hMLP, both mapped through a lightweight embedding layer and processed jointly with text. The model transcribes speech, follows spoken instructions, reasons over long recordings, and answers questions about charts and diagrams, optionally using a Python tool to zoom and crop images mid-reasoning. Scores like 91.4% on VoiceBench and 82.0% on CharXiv RQ with Python place it among the strongest open-weights multimodal models, though still behind Gemini 3.1 Pro.

    Epistemics: Calibration, Forecasting, and Censorship Resistance

    TML groups calibration, instruction following, and censorship resistance under the banner of epistemics. Calibration was trained with RL against proper scoring rules on resolved real-world questions, and it shows: Inkling’s ForecastBench Brier Index of 61.1 without search beats GPT-5.5 and Claude Opus 4.8, and its Prophet Arena score sits close to the frontier. Instruction following used two complementary automated graders, a rubric checklist and a claims grader that verifies factual assertions through agentic web search, so recall-spraying to hack rubrics gets penalized by the factuality check. Targeted abstention-aware QA datasets taught the model to say “I don’t know” or give hedged best guesses when appropriate, while still complying when a user demands a forced guess. Finally, the model was trained to answer directly on censorship-prone topics, with Cognition’s Propaganda and Censorship Eval finding strong non-compliance with censorship patterns.

    Safety for an Open-Weights Release

    Inkling was trained to an internal behavioral spec across all modalities and then checked by commissioned external safety testers. Evaluations covered dangerous capabilities (CBRN, cyber, loss of control) and human-AI threat vectors including sycophancy, vulnerable users, and harmful manipulation. On FORTRESS, which pairs adversarial harmful requests with benign look-alikes, Inkling posts the strongest adversarial score among the compared open models (78.0%) without collapsing on the benign side (95.9%), and it scores 98.6% on StrongREJECT. TML acknowledges the open question hanging over every open-weights release: how safety behavior holds up under fine-tuning, which it says it is actively studying on Tinker.

    Architecture and Training Recipe

    The MoE layout follows DeepSeek-V3: 256 routed experts and 2 shared experts per layer with 6 routed experts active per token, a sigmoid-based router, and auxiliary-loss-free load balancing. Attention interleaves sliding-window and global layers 5:1 with 8 KV heads, and positions are encoded with a learned relative positional embedding that TML found outperforms and out-extrapolates RoPE. Short convolutions appear after the key and value projections and on residual branch outputs. Optimization was hybrid, Muon for large matrices and Adam elsewhere, with hyperparameter schedules drawn from the lab’s modular manifolds research and weight decay coupled to the square of the learning rate to keep weight norms stable. Post-training bootstrapped from a small SFT phase on synthetic data from open models including Kimi K2.5, then spent the bulk of compute on large-scale RL. Everything ran on NVIDIA GB300 NVL72 systems.

    RL at Scale and the Emergent Compression of Thought

    TML scaled asynchronous RL past 30 million rollouts across two long continuous runs, with performance on a held-out aggregate of reasoning evals improving log-linearly the whole way. Along the way an unplanned behavior emerged: the chain of thought became progressively more concise, shedding grammatical overhead into a telegraphic style (“We need to understand” becomes “We need determine”) while remaining comprehensible and leaving final answers unaffected. No reward targeted this; token efficiency pressure alone drove the compression, echoing an observation Cognition made while training SWE-1.7. It is a vivid example of optimization discovering its own shorthand.

    Inkling-Small

    The preview of Inkling-Small is arguably the sleeper story: with 12B active parameters against Inkling’s 41B, it matches or exceeds the larger model on a surprising number of benchmarks, including GPQA Diamond (88.3% vs 87.2%), IFBench (83.4% vs 79.8%), and CharXiv RQ with Python (83.4% vs 82.0%). TML attributes this to pretraining data and recipe improvements made after the big model trained, with both models sharing the same post-training stack. The clearest gaps favoring big Inkling are factuality (SimpleQA 43.9% vs 20.9%), Terminal Bench, and Tau 3 Banking. Full weights for Inkling-Small will be released once testing finishes, and its cost and latency profile targets high-volume workloads like coding, LLM grading, and synthetic data generation.

    Availability and the Ecosystem Play

    Inkling is on Tinker today with 64K and 256K context options at a limited-time 50% discount, plus a free Inkling Playground chat interface with integrated web search in the Tinker console so developers can get a feel for the model before committing to a run. The cookbook gained native Inkling support and three new audio recipes, and a new tml-renderer handles chat templates, tool calls, reasoning content, and multimodal inputs. Deployment partnerships span Together, Fireworks, Modal, Databricks, and Baseten for APIs; RadixArk for SGLang and Miles; Inferact for vLLM; Lightseek for TokenSpeed; Unsloth for llama.cpp; and Hugging Face for transformers integration. Full weights are on Hugging Face in both the original checkpoint and an NVFP4 checkpoint for NVIDIA Blackwell inference.

    Notable Quotes

    “Our mission is to build AI that extends human will and judgment.”

    Thinking Machines Lab, opening the Inkling announcement

    The company’s north star, and the lens through which the whole release (customization, calibration, open weights) is framed.

    “Inkling is not the strongest overall model available today, open or closed. Instead, a combination of qualities makes it a good open-weights base for customization: multimodal capabilities, efficient thinking, and availability on Tinker for fine-tuning.”

    Thinking Machines Lab, positioning the release

    A rare piece of launch-day honesty from a frontier lab, and the strategic thesis of the whole release.

    “Picking the right base model to fine-tune is a qualitative judgment that combines measurable benchmarks with the unique feel of a model that comes from playing with it.”

    Thinking Machines Lab, on why the Inkling Playground exists

    An argument that vibes are data, from the lab that built a playground into a fine-tuning console.

    “Cost and latency are often binding constraints in real-world applications, and low latency in particular is crucial for enabling collaboration and improvement through iteration.”

    Thinking Machines Lab, on controllable thinking effort

    The case for evaluating models on their full effort-versus-performance curve instead of a single benchmark point.

    “A model that’s confident in every answer it gives, including when it’s missing info and confabulates, forces the user to double-check everything.”

    Thinking Machines Lab, on why calibration was a training target

    The clearest one-line justification for treating calibrated uncertainty as a capability rather than a nicety.

    “Together, the two graders improve helpfulness and reduce hallucination at the same time, rather than trading one for the other.”

    Thinking Machines Lab, on pairing a rubric grader with a web-searching claims grader

    A neat solution to rubric hacking: verify every claim with agentic search so spraying plausible facts stops paying.

    “Safety is crucial for open-weights models. We’re continuing to study safety behavior and capability uplift in customizable models, including how safety behavior is impacted by fine-tuning on Tinker.”

    Thinking Machines Lab, on the open question of fine-tunable safety

    The acknowledgment that safety trained into open weights must survive the very customization the product sells.

    “Inkling is just the start: our first release in a model family we will continue to build on.”

    Thinking Machines Lab, on the roadmap

    Together with the GB300 compute note, a clear signal that larger and stronger family members are coming.

    Read the full announcement, including the interactive demos, effort curves, and complete benchmark tables, on the Thinking Machines Lab blog.

    Related Reading

  • The Next 3 Years of AI, According to Steve Jurvetson: Moore’s Law, Superintelligence Odds, Elon Musk’s Operating Principles, and Where the Legendary SpaceX and Tesla Investor Is Betting Next

    Steve Jurvetson has spent 30 years funding the future before it was a category: an early check into SpaceX when space was not a venture sector, Tesla before electric cars were taken seriously, and now a portfolio spanning fusion, analog AI chips, and epigenetic editing at his firm Future Ventures. In this fireside chat he lays out what the next three years of AI actually look like, the three principles he has learned from working alongside Elon Musk for nearly three decades, the question he uses to separate missionary founders from opportunists, and why he thinks alignment of frontier AI systems may simply not be possible.

    TLDW

    Jurvetson argues the 130-year exponential in compute per dollar (Ray Kurzweil’s abstraction of Moore’s Law from his book The Age of Spiritual Machines) will keep running for at least three more years, carried by analog and custom AI silicon, and that this compounding is what makes startups and disruption possible at all. His gut says the next big leap will be “architecturally variant”: a new generation of labs going back to DeepMind’s founding premise of reinforcement learning, continuous learning, and novelty-seeking goal functions rather than bigger LLMs. He relays Anthropic co-founder Jack Clark’s 30 percent odds of superintelligence within a year but notes the crucial missing piece is that humans still set every goal. Adoption will be wildly uneven: anything made of atoms (cars, robots) switches over glacially, while creative work and white-collar categories like call centers (roughly 1 percent of US GDP) flip almost instantly. From Musk he draws three lessons: insane focus and saying no, maniacal attention to the cycle time of learning loops (Tesla gathers more AI training data every 4 days than Waymo has in its entire history), and being a magnet for talent by selling a grander mission. He explains Future Ventures’ current bets (fusion, free diagnostics via phone, slaughter-free meat, epigenetic editing, critical minerals, analog in-memory compute), tells solo founders their 30-day plan is to find a co-founder, predicts a turbulent transition to abundance, doubts Neuralink can keep pace with AI, dismisses Penrose’s quantum consciousness argument, and frames the post-work question with Man's Search for Meaning: humans need symbolic immortality, not just employment.

    Thoughts

    The most load-bearing claim in this conversation is not about scaling laws, it is about architecture. Jurvetson is telling you where the smart contrarian money is looking: away from ever-larger language models and back toward reinforcement learning agents with continuous learning and self-generated goals, the original DeepMind thesis that got shelved when LLMs took off. His framing of the open problem is unusually precise. The recursive self-improvement loops everyone is excited about are real, but every one of them is still human-directed. The goal-setting layer, what he calls the selection pressure of the evolutionary algorithm, is the “thin veneer of activity” AI does not yet do, and it happens to be the layer where superintelligence either does or does not arrive. That is a much sharper way to track AGI progress than benchmark scores: watch who cracks autonomous goal formation, not who tops a leaderboard.

    Almost everything else Jurvetson says reduces to a single metric: the cycle time of the learning loop. It is his explanation for Musk’s edge (launch cadence, the Tesla fleet as a data-collection machine), his filter for which industries flip fast (bits iterate at machine speed, atoms are stuck with 11-to-12-year car replacement cycles and FDA timelines), and even his bear case on Neuralink, which he has invested in. Biology cannot iterate at synthetic speed, so the substrate that learns fastest wins. Once you see the pattern, it becomes a genuinely useful lens for evaluating any company, career, or technology: ask how fast the loop spins, not how impressive the current artifact is.

    The aside that deserves the most attention is his flat statement that mechanistic interpretability will not bear fruit and that control and alignment of a cutting-edge system is not possible. His reasoning is structural, not rhetorical: anything produced by an iterative algorithm run billions of times (evolution, neural network training) is inherently inscrutable, and it will always be easier to build a new intelligence than to reverse engineer one you already made. He swaps “teenager” for “AI” whenever he thinks about control, which is funny until you notice he is one of the most connected investors in the Musk orbit saying the safety agenda rests on a false premise. Sitting that next to the 30 percent superintelligence odds he cites from Jack Clark produces an uncomfortable arithmetic that nobody on stage follows to its conclusion.

    For builders, the practical gold is the 50-year question. Ask a founder what their business looks like in 50 years: the opportunist laughs at the question, the missionary is relieved someone finally asked. Paired with his other filters (if only two out of ten people think your idea is crazy it is not bold enough, and a good business is one that could not have been started three years ago), it doubles as a hiring screen and a self-diagnostic. And his 30-day plan for a solo founder is refreshingly unglamorous: do not build the MVP, do not pitch investors, go persuade one person to give up their job and join you. If you cannot recruit a co-founder, that is the market’s first answer about your idea.

    Key Takeaways

    • Jurvetson invested early in SpaceX and Tesla precisely because space and automotive were not venture categories at all; a software-centric systems engineering approach applied to a sleepy industry that has not changed in decades unlocks enormous value, and that playbook is now rippling through every industry.
    • The Kurzweil curve plots 130 years of compute per dollar across five substrates (mechanical, relay, vacuum tube, discrete transistor, integrated circuit) and shows a 10,000 billion billion X improvement; Jurvetson calls it the most important thing ever graphed.
    • Customers buy compute capacity and memory, not transistors, and both have been “on rails” for 130 years; the default prediction for the next three years is simply that the curve keeps going.
    • When an incumbent declares Moore’s Law dead, it usually signals they are losing their business to someone new, as Intel was to Nvidia 15 years ago.
    • Analog chips and customized AI silicon that do discrete matrix multiply-and-add extremely efficiently will carry the mantle of Moore’s Law over the next three years.
    • Without exponential technological change there would be no startups: if business is predictable, the big get bigger and incumbents block new entrants; disruption is almost always computationally based.
    • Over the next three years AI ripples through energy, agriculture, and construction: three enormous industries that are growing as a percentage of GDP and are the least digitized on the planet, with healthcare close behind.
    • His gut says the next driver will be architecturally variant, possibly subsuming today’s models the way mixture of experts subsumes other architectures or massively parallel diffusion models reinterpret the transformer.
    • A whole new generation of neural labs is returning to the founding premise of DeepMind: reinforcement learning with continuous learning, let loose on the internet’s data sets, hunting for the algorithm that bootstraps intelligence.
    • The open question for these systems is the goal function: what plays the role of evolutionary selection pressure? Candidates include understanding the universe (the xAI mission) or a novelty-seeking algorithm that uses new discoveries as its measure of progress.
    • Jack Clark, co-founder of Anthropic, gives roughly 30 percent odds that superintelligence arrives within a year; Jurvetson declines to put odds on it himself and admits “I do not know” is the honest answer.
    • Today’s self-improving AI loops (automated verification, hyperparameter adjustment between training runs, AI-mediated experimentation) are real but still human-directed; goal setting remains the thin veneer AI does not do, and it may be the most important layer.
    • Human intelligence was bootstrapped on top of reactive limbic systems and emotional centers with cortex layered on top; it is an open philosophical question whether AI systems need to recapitulate that functional specialization to take on purpose and meaning.
    • Anything involving atoms switches over slowly: fully autonomous vehicles are inevitable (every car, train, and airplane), but people keep cars 11 to 12 years, so the physical swap-out cycle makes the transition feel glacial.
    • Physical robotics faces the same constraint: making a billion robots takes time even with recursive manufacturing techniques.
    • The domains that flip like wildfire are the ones we held as uniquely human: creative arts, moviemaking, and imagery came first, which Jurvetson finds somewhat shocking.
    • Call centers represent roughly 1 percent of US GDP and can switch over almost entirely and almost instantly; white-collar work generally has no physical swap-out cycle to slow it down.
    • People will increasingly prefer AI to human interactions when the AI is better: studies of physician bedside manner and customer service already show AIs doing a better job with emotional connection than humans.
    • Musk principle one is an insane ability to focus: running many companies forces ruthless prioritization, and he says no to anything that is not mission-critical right now, including a Craig Venter brainstorm on terraforming Mars because “none of this stuff on Mars matters” until Starship flies.
    • Musk principle two, the most important: maniacal focus on the cycle time of innovation, the core learning loop, whether launch cadence or fleet data; Tesla cameras gather more AI training data every 4 days than Waymo has collected in its entire history, because every vehicle collects data whether or not the customer paid for full self-driving.
    • Musk principle three: being a magnet for talent, screening for mastery by drilling into engineering crises a candidate actually solved rather than leaning on credentials (which are often an albatross), and framing the company as something grander (sustainable energy, multi-planetary humanity, understanding the universe) so the best people want to join.
    • Jurvetson filters founders with one question: what does your business look like in 50 years? Opportunists chuckle at the absurdity; missionaries are relieved and finally tell you what has been driving them all along. He passes on the ones who laugh.
    • The best startups hold two things in tension simultaneously: an audacious 50-to-500-year vision and a concrete plan to iterate with real customers over the next three years, chaining backward from the future to what must be built now.
    • The perpetual surprise of great companies is expanding option value: autonomous driving was nowhere in Tesla’s founding plan, and Starlink, direct-to-cell, and orbital data centers were not on SpaceX’s dance card even five years ago. Exploring the option space beats purposeful ten-year planning.
    • Future Ventures invests in things unlike anything they have seen before yet adjacent to what they know, ideally companies that are literally one of a kind.
    • Current bets include nuclear fusion and subcritical fusion that avoids NRC regulation, because energy is the third bottleneck for AI after talent and compute.
    • Other 500-year-problem bets: free healthcare via a cell phone (all diagnostics as a free global service, probably launching outside the US to bypass FDA and insurance), slaughter-free meat via cellular agriculture and mycelium, and construction, where labor productivity has been flat for 30 years.
    • Recent investments span epigenetic editing (the software of biology rather than the firmware of the genome, applied to crops, pesticides, and human health), critical minerals from deep sea mining to copper refining, and reshoring US industrial capacity.
    • Three separate analog AI chip investments approach the same goal from different angles, including Mythic’s in-memory compute doing 8-bit multiplication in a single transistor, each chasing 100X and then another 100X reduction in power per calculation.
    • The portfolio is roughly 40 percent life sciences and 60 percent IT, deliberately hunting the weird edge cases that fall through the cracks of traditional pharma VC: organ harvesting for transplant, a male birth control pill, dramatically improved IVF.
    • Old industries with no new entrants are the best targets: the four largest tunnel boring companies competing with the Boring Company were all started in the 1800s.
    • The 30-day plan for a single person with an idea: find a co-founder. Great startups tend to have a dynamic duo at the founding (Jobs and Wozniak, Sergey Brin and Larry Page, Larry Ellison and Bob Miner), and persuading one person to quit their job for your mission is the first real test of the idea.
    • A founding pair with diverse backgrounds and mutual respect sets the culture for everyone hired afterward and creates cognitive diversity that ripples through the whole firm.
    • Calibrate boldness by the crazy ratio: if 100 percent of people say your idea is crazy, take the feedback; nine out of ten is pretty good; if only two out of ten think it is crazy, it is not bold enough. Also ask whether the business could have been started three years ago; if yes, that is a bad sign.
    • Co-founders most often meet at universities, one of the few places where people cross academic disciplines; breakthrough innovation happens at the interstices between formally discrete fields, and LLMs are exceptionally good at exactly that cross-domain translation, opening a fountainhead of idea discovery.
    • Roughly 19 percent of global employment involves driving vehicles, and that work is going away, just more slowly than people imagine.
    • Humans have a fundamental desire for symbolic immortality: contributing something that outlasts our brief time here, whether children, books, philanthropy, or companies. Accumulated cultural knowledge, not biology, is the primary vector of human evolutionary progress.
    • There is no peaceful path from full employment to no employment: passing through 30, 40, 50 percent unemployment will be turbulent, and no politicians are taking a long-term perspective on it.
    • On Neuralink (which he invested in): expanding the sensory periphery is very doable (higher data rates, restoring hearing and spinal function, seeing more wavelengths), but upgrading core intelligence requires reverse engineering an inscrutable iterated system, and biology’s FDA-and-wetware timescales cannot keep up with synthetic learning loops.
    • Any product of an iterative algorithm run billions of times (evolution, neural networks, genetic programming) is inherently inscrutable; Jurvetson doubts mechanistic interpretability will bear fruit and does not think control or alignment of a cutting-edge AI system is possible, likening it to mind-controlling a teenager.
    • On Penrose’s quantum consciousness argument: there is no clear mechanism and no evidence of quantum processes in the brain, and arguments that consciousness requires our specific substrate are uncompelling; machines may one day have consciousness, just not necessarily human consciousness, the same way computer memory is real memory without being human memory.

    Detailed Summary

    Betting on Sectors That Do Not Exist Yet

    Asked what he saw in SpaceX that other investors missed, Jurvetson flips the question: there were almost no investors even considering space, just as automotive and nuclear energy were not venture sectors. The bet was on Elon Musk, whom he has known for 29 years and backed across all his companies (“and his cousins, too”), and on a thesis that has since crystallized: a software-centric systems engineering approach applied to a sleepy industry that has not changed in decades unlocks extraordinary value. Aerospace and automotive proved it, and the same conversion of industrial low-margin businesses into information businesses is now playing out across the economy.

    The 130-Year Compute Curve and the Next 3 Years

    Jurvetson polls the room on Kurzweil’s famous graph, first published around 1999, and finds only a quarter have seen what he calls the most important thing ever graphed: five successive technology substrates delivering a 10,000 billion billion X improvement in the computation a dollar buys, sustained over 130 years. Moore’s Law is just the most recent refraction of a longer, almost cosmological trend that transcends the dramas of individual companies. His baseline prediction for the next three years is that the curve keeps going, carried by analog chips and custom AI silicon optimized for matrix math, and he notes that when a company like Intel declares the end of Moore’s Law, it usually means they are losing to someone new, as they did to Nvidia. The deeper point: exponential technological change is the precondition for startups existing at all, because predictable business favors incumbents. AI is the most intense crucible of compute-centric innovation yet, and over the next three years it flows into energy, agriculture, construction, and healthcare, the largest and least digitized sectors.

    Architecturally Variant: The Return of Reinforcement Learning

    Pressed on what technology drives the next wave (better LLMs, world models, robotics), Jurvetson shares a gut feeling he stresses he has not yet invested in: something architecturally variant that may subsume today’s models. He points to a new generation of neural labs returning to DeepMind’s founding premise, reinforcement learning, which was set aside when LLMs took off. The open design problem is the goal function: what is the multi-decade agentic drive, the selection pressure, the definition of success beyond reproductive fitness? He floats understanding the universe (the Grok and xAI framing) and novelty-seeking algorithms that treat new discoveries as progress. The question these labs chase is whether a single reinforcement learning algorithm with continuous learning, let loose on the internet’s data, could bootstrap intelligence. He adds a caution about today’s chatbots: we ascribe consciousness and meaning where there is none. “There’s no light on inside,” at least for now.

    Superintelligence Odds and the Missing Goal-Setting Layer

    On whether self-directed, goal-setting AI arrives within three years, Jurvetson cites Jack Clark of Anthropic giving 30 percent odds of superintelligence next year, which he finds fun mostly because at least someone put a stake in the ground. The recursive self-improvement debate is live, but he insists on a distinction: the huge improvements in the current self-improving loop (automated verification, hyperparameter tuning between runs, AI-mediated experimentation) are all still directed by humans. Goal setting remains human, and while that may be only a thin veneer of remaining activity, it is arguably the most important part, and nobody is sure how the transition happens. It may require recapitulating the brain’s functional specialization, the limbic-then-cortex layering that produced our bootstrapped consciousness. His honest answer: he does not know and does not even have odds, because three years out is genuinely hard to predict.

    Atoms Move Slowly, Bits Sweep Like Wildfire

    The gap between what the technology can do and how we use it is governed by physics and replacement cycles. Fully autonomous vehicles are, to him, obviously inevitable for everything that moves on Earth, yet cars stay on the road 11 to 12 years, so the switchover feels glacial; a billion robots likewise take time to manufacture. What flips fast is the world of bits, and strangely it started with what we considered most human: creative arts, movies, and images. White-collar work follows because there is no physical swap-out cycle: call centers, about 1 percent of US GDP, can convert almost overnight. And people will increasingly prefer the AI when it is better, showing more emotional understanding and better reading of the situation, something already visible in comparisons of physician bedside manner and customer service quality.

    Three Principles from Working with Elon Musk

    Jurvetson opens with humility (even Maye Musk cannot explain how Elon became Elon, and the books piling up on his bedside table may not have been written by humans), but offers three observations from close range. First, an insane ability to focus. Running multiple companies paradoxically helps: nobody questions Elon skipping a holiday party, and he says no to fascinating distractions, including Jurvetson’s attempt to connect him with Craig Venter to brainstorm terraforming Mars with gene sequencers. Musk’s answer: none of it matters until Starship flies. Second, and even more important, a maniacal focus on the cycle time of innovation: how fast the core learning loop runs, whether launch cadence or fleet learning. The Tesla data flywheel is the exemplar: every car collects training data whether or not the owner paid for FSD, so Tesla gathers more data every 4 days than Waymo has in its history. Third, a well-honed talent stack: pattern recognition that ignores credentials (often an albatross), drills candidates on the engineering crises they actually navigated to test for real mastery, and wraps the company in a mission grand enough (sustainable energy, multi-planetary life, understanding the universe) that the best people want in, which compounds because great people attract great people.

    The 50-Year Question and Expanding Option Value

    How do founders stay true to a mission when 99 percent of the world says it is too early? Jurvetson admits selection bias: for 30 years he has tried to back only people with a sincere, almost messianic mission rather than arbitrage-seeking opportunists. His filter is to ask what the business looks like in 50 years. Opportunists laugh (“I’ll be on my third startup by then”); the best founders are relieved to finally unload the dream they have been hiding because “colonizing Mars is an uninvestable proposition” as a day-one pitch. The best startups pair an audacious 50-to-500-year vision with a plausible path of customer iteration over the next three years, chaining backward from the future. What still surprises him is how the option value of frontier companies keeps expanding: autonomous driving was not in Tesla’s founding plan at all, and SpaceX kept unfolding from cheap launch to Starlink to direct-to-cell to orbital data centers, none of which was on the dance card five years ago. Exploring the light cone of possibilities beats designing a ten-year plan.

    Where Future Ventures Is Betting Now

    The firm looks for companies unlike anything it has seen before yet adjacent to familiar ground, targeting problems that will obviously be solved 500 years from now. In energy: multiple fusion investments plus subcritical fusion that sidesteps NRC regulation, because energy is the third bottleneck for AI after people and compute. In health: free diagnostic healthcare delivered by cell phone as a global free service, likely launched outside the US to bypass FDA and reimbursement. In food: slaughter-free meat via cellular agriculture and mycelium. In construction: still looking, after trying and failing a few times in an industry where labor productivity has been flat for 30 years. Recent themes include epigenetic editing (the software of biology rather than the firmware of the genome, spanning crop health, pesticides, herbicides, and human health), critical minerals and metals from deep sea mining to copper refining as part of reshoring, and three separate analog AI chip bets, including Mythic’s in-memory compute doing 8-bit multiplication in a single transistor, each chasing successive 100X reductions in power per calculation. The mix runs about 40 percent life sciences, 60 percent IT, with a taste for the weird edge: organs grown for transplant, a male birth control pill, radically improved IVF. His favorite hunting ground is old, crappy industries with no new entrants, like tunnel boring, where the Boring Company’s four largest competitors were founded in the 1800s.

    Advice for Founders: Find Your Batman and Robin

    His 30-day plan for a single person with an idea is not an MVP or a pitch deck: find a co-founder. Startups tend to be founded by dynamic duos (Jobs and Wozniak, Sergey Brin and Larry Page, Larry Ellison and the lesser-known Bob Miner), and a pair with diverse backgrounds and mutual respect creates a rapid iteration loop and sets the cultural template for every future hire. Persuading one person to quit their job for your crazy idea is the first proof the mission can recruit. On calibrating craziness: if literally everyone thinks the idea is crazy, take the feedback; nine out of ten is pretty good; only two out of ten means it is not bold enough, because obvious ideas get done by others. Ask whether the business could have been started three years ago; the right answer is no. Co-founders most often meet at universities, where students (unlike professors in their stovepipes) cross-pollinate between academic disciplines, and breakthrough innovation lives at those interstices. As an aside, he notes LLMs excel at exactly this translation between domains, opening a new fountainhead of idea discovery we are only beginning to tap.

    When Machines Do Everything: Meaning, Abundance, and Turbulence

    Asked the closing question (when machines do everything, what is the meaning of life?), Jurvetson starts with scale: roughly 19 percent of global employment is driving vehicles, and it is going away. But humans want meaningful work, driven by what he calls a fundamental desire for symbolic immortality: children, books, philanthropy, companies named after founders, all instantiations of the urge to contribute something that outlasts us. Translating the question into humanity’s mission statement, he lands where Yuri Milner and Musk do: to understand the universe and add to accumulated knowledge, because culture, not biology, is the primary vector of human evolutionary progress. If we could hyperspace-jump to Peter Diamandis-style abundance, where everything physical costs a dollar a pound and machines do all labor, we could all be philosopher kings and artists. But he refuses to end on false comfort: there is no visible peaceful path from full employment through 30, 40, 50 percent unemployment, that transition will be turbulent, and no politicians are taking a long-term view of it.

    Neuralink, Inscrutable Systems, and the Alignment Heresy

    In audience Q&A, Jurvetson confirms he invested in Neuralink (the idea traces to the neural lace of Iain M. Banks’ novel Surface Detail, which he recommends) but offers a contrarian view. Working from the periphery is very promising: restoring broken function, fixing spinal cords, expanding senses, higher-bandwidth communication. Upgrading core functionality, actually making someone smarter, is another matter. His reasoning comes from decades of watching complex systems: any artifact produced by an iterative algorithm run billions of times (evolution, neural networks, genetic programming, cellular automata) is inherently inscrutable. That is why he doubts mechanistic interpretability will bear fruit and flatly does not think control and alignment are possible for a cutting-edge AI system; he mentally swaps “teenager” for “AI” whenever the control question comes up. The same inscrutability applies to the brain: it will be easier to build a new intelligence than to reverse engineer one already made, and FDA cycles plus human biology cannot iterate at the speed of synthetic learning loops, so he lacks faith Neuralink keeps up with AI. Kurzweil’s uploading dream, he suggests, is a case of wanting something to be true within one’s lifetime.

    Penrose, Quantum Brains, and Machine Consciousness

    On Roger Penrose’s argument that consciousness depends on quantum processes and is therefore unreachable by AI, Jurvetson is respectful of the man and dismissive of the claim: there is no clear mechanism (a speculative lithium isotope coupling aside), and it amounts to wishful thinking. Generalizing, he finds all vitalist arguments that our substrate is uniquely necessary uncompelling; you could make a better case that carbon is special to life than that neurons are essential to consciousness. His favorite reframe swaps in the word memory: computers have memory that is nothing like holographic, gracefully degrading human memory, yet nobody debates whether computer memory is real. Machines may likewise develop a different kind of consciousness without human consciousness. Declaring something impossible is a much higher-order proposition than admitting ignorance, so his position is: he does not know whether the current AI path leads to consciousness, but his gut says machines will get there one day, perhaps via evolution-like reinforcement learning approaches that recapitulate what biology already proved possible.

    Notable Quotes

    “I have this gut feeling that it’ll be something architecturally variant. It might subsume the models that we know now.”

    Steve Jurvetson, on what drives the next three years of AI

    “It’s almost cosmological. Like, why has humanity’s capacity to compute compounded for 130 years?”

    Steve Jurvetson, on the Kurzweil abstraction of Moore’s Law

    “If business is predictable, if there isn’t disruptive technological change, the big get bigger.”

    Steve Jurvetson, on why exponential compute is the precondition for startups

    “The Tesla cars today in their cameras gather for their AI training set more data every 4 days than Waymo has in its entire history.”

    Steve Jurvetson, on the data flywheel behind Musk’s learning-loop obsession

    “If it’s like only two people think it’s crazy, that’s bad because it’s clearly not bold enough. If it’s an obvious idea, other people will do it.”

    Steve Jurvetson, on calibrating how crazy a startup idea should be

    “Despite attempts at mechanistic interpretability in AI, I don’t think that’s going to bear fruit.”

    Steve Jurvetson, on why iterated systems are inherently inscrutable

    “It’d be easier to build a new intelligence than it is to reverse engineer one you’ve made.”

    Steve Jurvetson, on why he doubts Neuralink can keep pace with AI

    “I think all humans have a fundamental desire for symbolic immortality, this belief that we’ve contributed something to the world that transcends our brief time on this world.”

    Steve Jurvetson, on the meaning of life when machines do everything

    “It’s much higher order proposition to say something is impossible than to say I don’t know.”

    Steve Jurvetson, on whether AI can ever be conscious

    Watch the full conversation here: The Next 3 Years of AI: Lessons from Elon Musk’s First Investor.

    Related Reading

  • How GPT-5, Claude, and Gemini Are Actually Trained and Served: The Real Math Behind Frontier AI Infrastructure

    Reiner Pope, CEO of MatX and former TPU architect at Google, sat down with Dwarkesh Patel for a different kind of episode: a chalk-and-blackboard lecture on how frontier LLMs like GPT-5, Claude, and Gemini are actually trained and served. With nothing but a handful of equations and public API prices, Reiner reverse engineers an astonishing amount of what the labs are doing. If you have ever wondered why Fast Mode costs more, why context length stalls around 200k tokens, why models seem 100x over-trained, or why hyperscalers are pouring half a trillion dollars into memory, this is the most lucid explanation on the internet.

    TLDW

    Frontier LLM economics come down to two simple budgets: compute time and memory time. Once you write the rooflines on a blackboard, almost everything else falls out of them. Optimal batch size is roughly 300 times your sparsity ratio (around 2,000 to 3,000 tokens for a DeepSeek-style model). A new batch “train” departs every 20 milliseconds because that is how long it takes to read HBM end to end. Mixture of experts strongly favors staying inside a single rack, which is why scale-up domains went from 8 GPUs (Hopper) to 72 (Blackwell) to 500-plus (Rubin). Pipeline parallelism solves weight capacity but does nothing for KV cache, and adds painful per-hop latency, which is why Ilya famously said pipelining is not wise. Because of reinforcement learning and inference economics, frontier models are roughly 100x over-trained versus Chinchilla optimal, and a well-tuned model should output roughly as many tokens during deployment as went into its pre-training corpus. API prices leak the rest: Gemini’s 50% premium above 200k tokens reveals where KV memory time crosses weight memory time, prefill being 5x cheaper than decode confirms decode is memory bandwidth bound, and cache hit pricing tiers map directly to HBM, DDR, flash, and (yes) spinning disk. The lecture closes on a beautiful detour about the convergent evolution of neural nets and cryptographic ciphers.

    Key Takeaways

    • Two equations explain almost everything. A roofline analysis comparing compute time to memory fetch time predicts cost, latency, and architectural choices with shocking accuracy.
    • Optimal batch size is about 300 times sparsity. For a DeepSeek model that activates 32 of 256 experts, that lands around 2,000 to 3,000 tokens per batch. Real deployments go a bit higher to leave headroom.
    • The 20 millisecond train. A new batch departs every 20ms because that is how long it takes to read all of HBM once. Worst-case queue latency is roughly 40ms.
    • Fast Mode is just smaller batches. Pay 6x more, get 2.5x faster decode by amortizing weights over fewer users. There is a hard latency floor at the HBM read time.
    • Slow Mode would not save much. Once you are past the optimal batch size, the cost-per-token plateau is dominated by compute, not weight fetches. You cannot meaningfully amortize KV cache because it is unique per sequence.
    • One rack is the natural MoE unit. Expert parallelism wants all-to-all communication, which strongly favors the scale-up network (NVLink) over the scale-out network (roughly 8x slower).
    • Bigger scale-up domains drove model scaling. The jump from 8 (Hopper) to 72 (Blackwell) to 500-plus (Rubin) GPUs per rack increased aggregate memory bandwidth by 8x, which is why trillion-plus parameter models only became viable recently.
    • Pipeline parallelism is overrated for inference. It saves on weight memory capacity but does nothing for KV cache memory. It also adds milliseconds of latency per hop in decode.
    • Why Ilya said pipelining is not wise. Architectural constraints (cross-layer residuals like in Kimi) and the inability to amortize weight loads across micro-batches make pipelining a hassle in training too.
    • The memory wall is real and paradoxical. Hyperscalers reportedly spend 50% of CapEx on memory, yet racks have far more HBM than a trillion-parameter model needs. The capacity is there for KV cache and batch size, not for weights.
    • Frontier models are roughly 100x over-trained vs Chinchilla. When you minimize total cost across pre-training plus RL plus inference, smaller models trained on more data win.
    • Each model should output roughly all human knowledge. If you equalize pre-training and inference compute, the total tokens served by a model during its lifetime should approximate its training corpus. Roughly 150 trillion in, 150 trillion out.
    • API pricing reveals architecture. Gemini’s 50% premium above 200k context, the 5x decode-vs-prefill ratio, and cache duration tiers all leak detailed information about KV size, memory bottlenecks, and storage hierarchy.
    • KV cache is roughly 2KB per token. Solving Gemini’s pricing equation gives a plausible 1.6 to 2 kilobytes per token at 100B active parameters and 200k context.
    • Decode is memory bandwidth bound, prefill is compute bound. The 5x price gap is direct evidence.
    • Cache pricing maps to memory tiers. The 5-minute and 1-hour cache durations probably correspond to flash and spinning disk drain times respectively. LLM serving uses spinning disk.
    • Context length is stuck near 200k. Memory bandwidth, not compute, is the binding constraint. Sparse attention gives a square-root improvement but is not infinite.
    • Cryptography and neural nets are mathematical cousins. Both rely on jumbling information across inputs. Feistel ciphers led directly to RevNets (reversible neural networks). Adversarial attacks mirror the cipher avalanche property.

    Detailed Summary

    The Roofline: Compute Time vs Memory Time

    Reiner starts with the simplest possible model of LLM inference. The time to do a forward pass is bounded below by the maximum of compute time and memory fetch time. Compute time is the batch size times active parameters divided by FLOPs. Memory time is total parameters divided by memory bandwidth, plus a KV cache term that scales with batch size and context length. From these two equations, almost every economic and architectural fact about modern LLMs can be derived.

    Plotting cost per token against batch size gives a clean picture: at low batch you pay enormous overhead because you cannot amortize the weight fetches, and at high batch you hit a compute floor. There is a sweet spot where memory bandwidth time equals compute time. That sweet spot is what Fast Mode and Slow Mode are tuning around.

    Why Fast Mode Costs More: The Batch Trade-Off

    When Claude Code or Codex offers Fast Mode at 6x the price for 2.5x the speed, what is really happening is that they are running you at a smaller batch size. Smaller batch means weight loads are amortized over fewer users, so cost per token goes up. But latency goes down because each forward pass touches less data. There is a hard floor on latency because you have to read every byte of HBM at least once per token, and that takes about 20 milliseconds on Blackwell-class hardware. There is also a soft ceiling on Slow Mode savings because the unamortizable parts (KV cache fetches, compute) eventually dominate.

    The 20 Millisecond Train

    HBM capacity divided by HBM bandwidth lands consistently around 20 milliseconds across generations of Nvidia hardware. That is the natural cadence at which a frontier model can run a forward pass over all its weights. Reiner uses a memorable analogy: a train departs every 20 milliseconds. Any users whose requests are ready board the train. If the train is full, they wait. If it is empty, it leaves anyway. This is why you do not need millions of concurrent users to saturate a model’s batch. You only need enough to fill a 2,000-token train every 20ms.

    Why Optimal Batch Size Is About 300 Times Sparsity

    Setting compute time equal to weight fetch time and rearranging gives a beautiful result: batch size needs to be greater than (FLOPs / memory bandwidth) times (total params / active params). The hardware ratio is a dimensionless 300 on most GPUs and has stayed remarkably stable from A100 through Hopper, Blackwell, and Rubin. The model term is just the sparsity ratio. For DeepSeek with 32 of 256 experts active, that is 8. So optimal batch is around 2,400 tokens. Real deployments push this to 3x to leave headroom for non-ideal efficiency. At 64 trains per second, that is roughly 128,000 tokens per second per replica, or about 1/1000 of Gemini’s reported global throughput.

    Mixture of Experts Wants to Live Inside a Rack

    MoE all-to-all routing means every token can be sent to any expert on any GPU. The communication pattern strongly prefers the fast scale-up network (NVLink) inside a rack to the slower scale-out network between racks. Scale-out is roughly 8x slower in bandwidth. This is why one rack ends up being the natural unit for an expert layer, and why Nvidia’s progression from 8 GPUs per rack (Hopper) to 72 (Blackwell) to 500-plus (Rubin) has been such a big deal for model size scaling.

    Reiner walks through the physical constraints: cable density, bend radius, weight, power, cooling. Modern racks are pushing every dimension to the limit. Stuffing more GPUs into the scale-up domain is genuinely a hardware engineering problem.

    Pipeline Parallelism: Why Ilya Said It Is Not Wise

    Pipelining splits model layers across racks. It is the natural way to scale beyond the scale-up domain for very large models. But it has problems. In inference, pipelining does not save runtime, it only saves memory capacity per rack, which already is not the binding constraint because trillion-parameter models only need a terabyte and racks have 10x that. In training, pipelining creates the famous bubble (idle GPU time at the start and end of each pipeline pass) and forces micro-batching, which kills your ability to amortize weight loads across the global batch.

    There is also an architectural cost. Models like Kimi use cross-layer residual connections where attention attends to layers a few back, and pipelining makes those patterns very hard to implement cleanly. Ilya’s quip “as we now know, pipelining is not wise” captures all of this.

    The Memory Wall Paradox

    Industry analysts report that hyperscalers are spending 50% of CapEx on memory this year, while smartphones and laptops are seeing 30% volume drops because there is not enough HBM and DDR to go around. Yet a Blackwell rack already has tens of terabytes of HBM, far more than a trillion-parameter model needs. The reason is that all that extra capacity goes to KV cache, batch size, and longer context. The bandwidth, not the capacity, is what matters most for weight loading. This also implies that hardware could be designed with less HBM per GPU if you commit to pipelining the weights, which is a real architectural option for a chip startup like MatX.

    Reinforcement Learning and the 100x Over-Training of Frontier Models

    Chinchilla scaling laws say a model with N active parameters should be trained on roughly 20N tokens for compute-optimal training. But frontier labs do not just minimize training cost. They minimize training plus inference cost across the model’s deployment lifetime. With reinforcement learning added to the mix, the cost equation has three terms: pre-training (6 times active params times tokens), RL (somewhere between 2x and 6x times active params times RL tokens, with a 30% efficiency penalty for decode-heavy rollouts), and inference (2 times active params times inference tokens).

    If you assume those three roughly equalize at the optimum (a heuristic that holds for many cost curves), you get a clean conclusion: the data going into pre-training should be roughly equal to the data going into RL, which should be roughly equal to the tokens served at inference. With 100 billion active parameters and roughly 150 trillion training tokens, that is about 75x past Chinchilla optimal. Reiner rounds it to 100x. This is the most concrete first-principles argument for why frontier models are so deeply over-trained, and it implies that as inference traffic grows, models should keep getting smaller and longer-trained.

    Each Model Should Output All of Human Knowledge

    The most jaw-dropping consequence: if you equalize pre-training and inference compute, then the total tokens generated by a model across its deployment lifetime should approximate the size of its training corpus. GPT-5, served to hundreds of millions of users for two months, will collectively output something on the order of 150 trillion tokens. That is roughly the sum of human knowledge in textual form. Each frontier model is, in this sense, a one-shot universal author of a corpus the size of its source material.

    API Prices Leak Architecture

    This is where the lecture gets really fun. Gemini 3.1 charges 50% more for context above 200k tokens. Setting memory time equal to compute time at exactly 200k context and solving for KV cache size gives roughly 1.6 to 2 kilobytes per token, which is plausible for a model with 8 KV heads, dense attention, and head dimension of 128.

    The 5x premium for output (decode) tokens versus input (prefill) tokens is direct evidence that decode is severely memory bandwidth bound and prefill is compute bound. Prefill processes many tokens per weight load, so it amortizes memory cost over the whole sequence. Decode processes one token per weight load, so it pays full memory cost every time.

    Cache hits priced at one tenth of cache misses tell you that storing the KV cache in HBM (or DDR or flash) is much cheaper than recomputing it from scratch. The two cache duration tiers (5 minutes and 1 hour) probably correspond to memory tiers whose drain times match those durations: flash for the 5-minute tier, spinning disk for the 1-hour tier. Yes, spinning disk is in the modern LLM serving stack, despite being decades-old technology.

    Why Context Length Has Plateaued at 200k

    Context lengths shot up from 8k to roughly 200k during the GPT-3 to GPT-4 era and have stayed roughly flat for the past two years. Reiner argues this is the natural balance point where memory bandwidth cost crosses compute cost. Going to a million tokens is expensive. Going to 100 million tokens (which Dario has hinted is needed for true continual learning via in-context learning) is essentially impossible without either a memory technology breakthrough or a much more aggressive sparse attention scheme. Sparse attention helps with a square-root improvement, but it is not unlimited. Going too sparse trades off too much quality.

    Cryptography Meets Neural Nets

    The episode ends with a lovely intellectual detour. Cryptographic protocols and transformer architectures both rely on jumbling information across all inputs. They are doing inverse versions of the same operation: ciphers take structured input and produce randomness, while neural nets take noisy input and extract structure. Both fields use differentiation as their primary attack vector (differential cryptanalysis on ciphers, gradient descent on neural nets). Adversarial attacks on image classifiers exploit exactly the avalanche property that good ciphers are designed for.

    The most concrete crossover: Feistel ciphers, which let you build invertible functions out of non-invertible ones, were ported into deep learning as RevNets (reversible networks) in 2017. RevNets let you run the entire network backwards during the backward pass, eliminating the need to store activations and dramatically reducing training memory footprint. It is the opposite trade-off of KV caching: spending compute to save memory rather than spending memory to save compute.

    Thoughts

    The most striking thing about this episode is how much can be deduced from a few equations and the public API price sheets of the major labs. The labs treat their architectures as trade secrets, but the moment they price tokens to be close to cost (which competition forces them to do), the prices themselves leak the underlying ratios. Anyone with a pen and paper can reverse engineer the KV cache size, the memory tier hierarchy, and the compute-vs-memory bottleneck profile of a frontier model. There is a lesson here for builders: in competitive markets, the prices tell you almost everything.

    The 100x over-training result has interesting implications for what comes next. If the optimal balance shifts further toward inference (as adoption keeps growing), models should get smaller and longer-trained. That is good news for serving costs and bad news for training-compute-as-moat. The biggest determinant of model quality might increasingly be data quality and RL environment design, not raw pre-training compute. This squares with what is visible publicly: the leading labs are investing heavily in RL infrastructure, evaluations, and synthetic data pipelines.

    The memory wall is the most underrated infrastructure story in AI. Most people think of compute as the bottleneck, but Reiner makes it clear that memory bandwidth is what actually limits context length, which limits how agentic a model can be in practice. If you cannot get to 100 million token contexts, you probably cannot have an AI agent that has been working with you for a month and remembers everything. Either some sparse attention scheme has to give us cheap effective context length, or we need a memory hardware breakthrough, or we have to invent some form of continual learning that does not rely on context windows. None of those paths are obviously easy, and the fact that context length has been flat for two years despite enormous investment suggests we are stuck against a real wall.

    The cryptography parallel is the kind of cross-disciplinary insight that does not show up enough in AI discourse. Treating neural networks as a kind of differentiable cipher reframes a lot of the architecture choices (residual connections, layer normalization, attention) as deliberate efforts to make the function smooth and invertible enough to learn, in contrast to ciphers, which are deliberately designed to resist exactly that. Adversarial robustness research probably has a lot more to learn from cryptanalysis than it currently does.

    Finally, the format itself is a win. Most AI podcasts are conversational, which is great for personality but bad for technical depth. A blackboard lecture with an interlocutor who asks naive questions at the right moments is a much higher bandwidth medium. More of this, please.

  • Composer: Building a Fast Frontier Model with Reinforcement Learning

    Composer represents Cursor’s most ambitious step yet toward a new generation of intelligent, high-speed coding agents. Built through deep reinforcement learning (RL) and large-scale infrastructure, Composer delivers frontier-level results at speeds up to four times faster than comparable models:contentReference[oaicite:0]{index=0}. It isn’t just another large language model; it’s an actively trained software engineering assistant optimized to think, plan, and code with precision — in real time.

    From Cheetah to Composer: The Evolution of Speed

    The origins of Composer go back to an experimental prototype called Cheetah, an agent Cursor developed to study how much faster coding models could get before hitting usability limits. Developers consistently preferred the speed and fluidity of an agent that responded instantly, keeping them “in flow.” Cheetah proved the concept, but it was Composer that matured it — integrating reinforcement learning and mixture-of-experts (MoE) architecture to achieve both speed and intelligence.

    Composer’s training goal was simple but demanding: make the model capable of solving real-world programming challenges in real codebases using actual developer tools. During RL, Composer was given tasks like editing files, running terminal commands, performing semantic searches, or refactoring code. Its objective wasn’t just to get the right answer — it was to work efficiently, using minimal steps, adhering to existing abstractions, and maintaining code quality:contentReference[oaicite:1]{index=1}.

    Training on Real Engineering Environments

    Rather than relying on synthetic datasets or static benchmarks, Cursor trained Composer within a dynamic software environment. Every RL episode simulated an authentic engineering workflow — debugging, writing unit tests, applying linter fixes, and performing large-scale refactors. Over time, Composer developed behaviors that mirror an experienced developer’s workflow. It learned when to open a file, when to search globally, and when to execute a command rather than speculate.

    Cursor’s evaluation framework, Cursor Bench, measures progress by realism rather than abstract metrics. It compiles actual agent requests from engineers and compares Composer’s solutions to human-curated optimal responses. This lets Cursor measure not just correctness, but also how well the model respects a team’s architecture, naming conventions, and software practices — metrics that matter in production environments.

    Reinforcement Learning as a Performance Engine

    Reinforcement learning is at the heart of Composer’s performance. Unlike supervised fine-tuning, which simply mimics examples, RL rewards Composer for producing high-quality, efficient, and contextually relevant work. It actively learns to choose the right tools, minimize unnecessary output, and exploit parallelism across tasks. The model was even rewarded for avoiding unsupported claims — pushing it to generate more verifiable and responsible code suggestions.

    As RL progressed, emergent behaviors appeared. Composer began autonomously running semantic searches to explore codebases, fixing linter errors, and even generating and executing tests to validate its own work. These self-taught habits transformed it from a passive text generator into an active agent capable of iterative reasoning.

    Infrastructure at Scale: Thousands of Sandboxed Agents

    Behind Composer’s intelligence is a massive engineering effort. Training large MoE models efficiently requires significant parallelization and precision management. Cursor’s infrastructure, built with PyTorch and Ray, powers asynchronous RL at scale. Their system supports thousands of simultaneous environments, each a sandboxed virtual workspace where Composer experiments safely with file edits, code execution, and search queries.

    To achieve this scale, the team integrated MXFP8 MoE kernels with expert and hybrid-sharded data parallelism. This setup allows distributed training across thousands of NVIDIA GPUs with minimal communication cost — effectively combining speed, scale, and precision. MXFP8 also enables faster inference without any need for post-training quantization, giving developers real-world performance gains instantly.

    Cursor’s infrastructure can spawn hundreds of thousands of concurrent sandboxed coding environments. This capability, adapted from their Background Agents system, was essential to unify RL experiments with production-grade conditions. It ensures that Composer’s training environment matches the complexity of real-world coding, creating a model genuinely optimized for developer workflows.

    The Cursor Bench and What “Frontier” Means

    Composer’s benchmark performance earned it a place in what Cursor calls the “Fast Frontier” class — models designed for efficient inference while maintaining top-tier quality. This group includes systems like Haiku 4.5 and Gemini Flash 2.5. While GPT-5 and Sonnet 4.5 remain the strongest overall, Composer outperforms nearly every open-weight model, including Qwen Coder and GLM 4.6:contentReference[oaicite:2]{index=2}. In tokens-per-second performance, Composer’s throughput is among the highest ever measured under the standardized Anthropic tokenizer.

    Built by Developers, for Developers

    Composer isn’t just research — it’s in daily use inside Cursor. Engineers rely on it for their own development, using it to edit code, manage large repositories, and explore unfamiliar projects. This internal dogfooding loop means Composer is constantly tested and improved in real production contexts. Its success is measured by one thing: whether it helps developers get more done, faster, and with fewer interruptions.

    Cursor’s goal isn’t to replace developers, but to enhance them — providing an assistant that acts as an extension of their workflow. By combining fast inference, contextual understanding, and reinforcement learning, Composer turns AI from a static completion tool into a real collaborator.

    Wrap Up

    Composer represents a milestone in AI-assisted software engineering. It demonstrates that reinforcement learning, when applied at scale with the right infrastructure and metrics, can produce agents that are not only faster but also more disciplined, efficient, and trustworthy. For developers, it’s a step toward a future where coding feels as seamless and interactive as conversation — powered by an agent that truly understands how to build software.

  • Revolutionizing AI: How the Mixture of Experts Model is Changing Machine Learning

    Revolutionizing AI: How the Mixture of Experts Model is Changing Machine Learning

    The world of artificial intelligence is witnessing a paradigm shift with the emergence of the Mixture of Experts (MoE) model, a cutting-edge machine learning architecture. This innovative approach leverages the power of multiple specialized models, each adept at handling different segments of the data spectrum, to tackle complex problems more efficiently than ever before.

    1. The Ensemble of Specialized Models: At the heart of the MoE model lies the concept of multiple expert models. Each expert, typically a neural network, is meticulously trained to excel in a specific subset of data. This structure mirrors a team of specialists, where each member brings their unique expertise to solve intricate problems.

    2. The Strategic Gating Network: An integral part of this architecture is the gating network. This network acts as a strategic allocator, determining the contribution level of each expert for a given input. It assigns weights to their outputs, identifying which experts are most relevant for a particular case.

    3. Synchronized Training: A pivotal phase in the MoE model is the training period, where the expert networks and the gating network are trained in tandem. The gating network masters the art of distributing input data to the most suitable experts, while the experts fine-tune their skills for their designated data subsets.

    4. Unmatched Advantages: The MoE model shines in scenarios where the input space exhibits diverse characteristics. By segmenting the problem, it demonstrates exceptional efficiency in handling complex, high-dimensional data, outperforming traditional monolithic models.

    5. Scalability and Parallel Processing: Tailor-made for parallel processing, MoE architectures excel in scalability. Each expert can be independently trained on different data segments, making the model highly efficient for extensive datasets.

    6. Diverse Applications: The practicality of MoE models is evident across various domains, including language modeling, image recognition, and recommendation systems. These fields often require specialized handling for different data types, a task perfectly suited for the MoE approach.

    In essence, the Mixture of Experts model signifies a significant leap in machine learning. By combining the strengths of specialized models, it offers a more effective solution for complex tasks, marking a shift towards more modular and adaptable AI architectures.