Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper

The guest frames his entire company mission as driving the price of tokens to the floor using every layer of the stack.

“My job is to make the tokens as cheap as”

assertion · guest

unverifiable · high confidence — A statement of personal mission, not a factual assertion. The 'cost floor' positioning is the standard framing of every inference-serving company from Together and Fireworks to DeepInfra; nothing here can be true or false.

To check: Nothing to check; at most, whether company behaviour (chip purchasing, siting) matches the stated mission over time.

His company claims it serves open-source model tokens at prices no competitor can match — the company's own article of faith.

“will serve those tokens to them at a price that is unbeatable in the market”

assertion · guest

unverifiable · high confidence — Self-description with no benchmark. Open-weight token pricing is publicly observable and fiercely competitive (OpenRouter, Together, Fireworks, DeepInfra, Novita, plus DeepSeek's own API), so 'unbeatable' is checkable in principle but he offers no figure and price leadership on any given model rotates weekly.

To check: OpenRouter or Artificial Analysis price-per-million-token leaderboards for the specific open models served, compared like-for-like on quantisation and throughput tier.

A 10x cost reduction in any good creates a new product category, and the guest aspires to do this for tokens.

“whenever you make something 10 times cheaper, it's a new product category”

assertion · guest

plausible · medium confidence — A very widely circulated heuristic in tech investing (Bezos, a16z, Wright's-law arguments), with real supporting cases — genome sequencing, LED lighting, cloud vs. on-prem, SpaceX launch cost. It is a rule of thumb rather than a measured regularity, and counterexamples exist where 10x cheaper just deflated an existing category.

To check: No single settling evidence; the closest is a systematic survey of goods that fell 10x in price and whether new categories followed.

Cost per token is the guest's explicit optimization target, though he doubts tokens are the final unit of work.

“my north star is I want to have the lowest cost per token in the industry and do that by a mile”

assertion · guest

unverifiable · high confidence — Statement of intent. Worth noting he immediately hedges that tokens may not be the right unit of work, which is a live debate — outcome-based and task-based pricing are being trialled by several agent vendors.

To check: Whether the company's published price list actually leads the market by a wide margin at comparable quality.

The guest asserts agentic inference will be dominated by long-horizon tasks running for hours or days, not low-latency interaction.

“the future of agentic inference is long horizon tasks”

assertion · guest

plausible · medium confidence — This tracks the METR finding that the task length models can complete at 50% success has been doubling roughly every seven months, and matches the visible product shift toward async coding agents (Codex cloud tasks, Claude Code background runs, Devin). It remains a directional bet, not a measured fact, and the strong form ('the future') skips over the enormous current volume of interactive chat.

To check: METR-style time-horizon evaluations over subsequent model releases, plus provider-reported split of batch/async vs. synchronous API traffic.

Work done proactively in the background beats fast responses, because the user never waits at all.

“the best latency is no latency at all”

assertion · guest

plausible · high confidence — A rhetorical framing of the well-known async/batch principle: work precomputed before the user asks has zero perceived wait. True by construction where the system can correctly anticipate the request; the hard part he skips is prediction accuracy and the waste of speculative work that is never used.

To check: Product telemetry on how often proactively generated work is actually consumed versus discarded.

The guest dates practical test-time-compute scaling to Opus 4.5, which he calls the first model usable for long-horizon agent work.

“Opus45 was the first agent that was at all suitable for longer horizon tasks”

assertion · guest

contested · medium confidence — Claude Opus 4.5 (late Nov 2025) did post a notable jump on agentic and long-horizon coding benchmarks, so the sentiment is defensible. But 'first at all suitable' is a judgement many practitioners would dispute in favour of o3/o1-style reasoning models, GPT-5-Codex, or Claude Sonnet 4 with extended thinking, all of which were already running multi-hour agent loops. My knowledge of the period right around and after that release is thin.

To check: METR 50%/80% time-horizon numbers and SWE-bench-Verified / Terminal-Bench agentic scores for Opus 4.5 versus its immediate predecessors and o-series contemporaries.

Current agents reliably sustain roughly one-hour task runs, not days.

“are capable of running for an hour at a time. I wouldn't say it's days but definitely an hour is quite suitable today”

quantity · guest

consistent · medium confidence — Matches the published METR horizon estimates for late-2025 frontier models — roughly two hours at 50% reliability and well under an hour at 80% reliability — so 'an hour, not days' is a fair characterisation, and his refusal to claim days is the honest part. Note the numbers are reliability-threshold dependent and he doesn't say which he means.

To check: METR time-horizon curves for the models in question, at both the 50% and 80% success thresholds.

The guest predicts inference workloads shift from a 50/50 background/real-time split this year to 90/10 in favour of background.

“to end this year at maybe 50/50 background and uh and real-time workloads, but I see this going to 9010 in favor of background”

prediction · guest

unverifiable · high confidence — A forecast, and one about a quantity nobody publishes consistently — there is no accepted definition or industry measure of 'background vs. real-time' token share. His underlying argument (human attention caps interactive consumption, background is unbounded) is sound and is the same reasoning behind batch-API discounts.

To check: Provider disclosure of batch/async versus synchronous token volumes, or an aggregate like OpenRouter's traffic mix broken out by request type.

Software security is becoming a function of how much inference spend was thrown at attacking it.

“really a question of how many dollars did you spend on anthropics APIs trying to break into your software. uh that is the best indication for how secure it is”

assertion · guest

plausible · medium confidence — He flags it as a community joke, which is the right register. It rhymes with real developments — Google's Big Sleep finding real CVEs, XBOW topping HackerOne leaderboards, DARPA AIxCC — where discovery scales with compute spent. As a security metric it is obviously incomplete: spend without coverage or good harnesses proves little.

To check: Published studies correlating inference spend on automated pentesting with subsequently discovered vulnerability counts, e.g. AIxCC results or XBOW disclosure data.

Model capability is jagged: small models find bugs large models miss and vice versa, so diverse sampling beats using only the best model.

“the frontier of intelligence here is quite jagged”

assertion · guest

consistent · high confidence — 'Jagged frontier' is the accepted term since the Dell'Acqua et al. HBS/BCG study, and the specific mechanism he describes — smaller or different models surfacing bugs the biggest model misses, so diverse sampling beats best-of-one — is exactly the pass@k / ensemble-diversity result seen throughout the fuzzing and code-repair literature.

To check: Union-vs-intersection bug counts across model sizes on a fixed vulnerability benchmark such as ARVO or OSS-Fuzz-derived sets.

The dollar cost of a definitive answer to a research question is already in the thousands and could fall to tens of dollars.

“It's not millions, it's thousands and maybe it could be hundreds or even tens of dollars in the near future to have a definitive answer to any scientific question”

prediction · guest

unverifiable · high confidence — A prediction resting on an undefined term — 'definitive answer to any scientific question' — that no cost figure can settle. Deep-research products do cost single-digit-to-hundreds of dollars per run today, but 'definitive' answers to open scientific questions are a category the current evidence (GPQA, literature-synthesis evals, AI co-scientist pilots) does not reach.

To check: A benchmark of open research questions with expert adjudication of answer sufficiency, priced per run.

Non-verifiable domains — human taste, writing quality, art — remain unsolved and possibly unsolvable, so his company avoids them.

“human taste into that category we have not solved human taste yet and I don't know that it fundamentally can be”

assertion · guest

contested · medium confidence — The verifiable/non-verifiable split is the standard framing for where RL post-training works (cf. Jason Wei's 'verifier's law'), and his hedge is appropriate. But 'possibly unsolvable' is a strong minority position: reward models trained on human preference already capture a great deal of taste, and models routinely win blind creative-writing preference tests. The realistic disagreement is about ceiling and reward-hacking, not impossibility.

To check: Blind expert-panel evaluations of model versus human output in taste-heavy domains tracked over successive model generations.

GPUs cannot be simultaneously throughput-maximal and latency-minimal; batching raises utilisation but lengthens each request's residence time.

“There's a fundamental trade-off on the GPU between being uh throughput oriented or latency optimized”

assertion · guest

consistent · high confidence — Textbook roofline and queuing theory. Larger batches raise arithmetic intensity and tokens/sec/GPU but increase per-request residence time; the TTFT/TPOT-versus-throughput Pareto curve is exactly what vLLM, SGLang and TensorRT-LLM benchmark papers publish. His bus-versus-car analogy is standard.

To check: Any published throughput-versus-latency Pareto plot from vLLM/SGLang benchmarks at varying batch sizes.

The guest predicts the dominant AI usage shape flips from interactive chatbots to background agents within a year.

“that's the most profound change we're going to see in the next year. We're going to move away from chatbots to more proactive or background agents”

prediction · guest

unverifiable · high confidence — A dated forecast. Directionally it aligns with observable product moves (async cloud agents, scheduled tasks, agent SDKs), but 'move away from chatbots' within a year is a stronger claim than the trend supports — ChatGPT-style interactive use remains the overwhelming volume driver by user count.

To check: A year passing, then comparing MAU/token-share of interactive chat products versus async agent products.

Sharding a matrix multiply across eight GPUs yields only four to five times the speed due to communication overhead and smaller tiles.

“It's sublinear scaling. You'll use eight times more hardware, but you won't get eight times the speed. You might get like four to fivex the speed.”

quantity · guest

consistent · high confidence — Standard tensor-parallel scaling behaviour: all-reduce communication per layer plus reduced GEMM tile efficiency at smaller shards typically yields 50–70% scaling efficiency at TP=8 within an NVLink domain, i.e. roughly 4–5.5x. His attribution to communication overhead and tile size is the right mechanism.

To check: TP=1/2/4/8 latency scaling curves from TensorRT-LLM or vLLM on a single NVL8 or NVL72 node.

NVLink is a latency technology and is required for low-latency inference, which is why non-Nvidia chips can still serve throughput workloads.

“NVLink is mandatory I would say for low latency inference”

assertion · guest

contested · medium confidence — True as stated about GPU tensor parallelism — PCIe-only TP collapses on latency-critical all-reduces. But 'mandatory for low-latency inference' full stop is contradicted by Groq and Cerebras, which post the lowest tokens/sec latencies in the market without NVLink, and by TPU pods using ICI. He softens it later ('Nvidia is basically mandatory for tensor parallelism'), which is the accurate version.

To check: Artificial Analysis latency leaderboards showing non-NVLink architectures (Groq, Cerebras, TPU) at or above NVLink-based latency.

Blackwell pairs 288GB of stacked HBM with roughly 500MB of on-die SRAM.

“Blackwell has 288 GB of HPM capacity around the logic die. And the logic die itself maybe only has like 500 megabytes of of SRAM”

quantity · guest

consistent · medium confidence — 288GB HBM3E is correct for Blackwell Ultra (B300/GB300); the earlier B200 is 192GB. The SRAM figure looks generous to me: summing register files, L1/shared and L2 across both dies lands nearer 250–300MB by my reckoning, though it is the right order of magnitude and his separate remark that a full 800mm² SRAM die would be single-digit gigabytes is about right for N4/N5 SRAM density.

To check: NVIDIA Blackwell architecture whitepaper for HBM capacity, and per-SM register file / L1 / L2 totals for the B200/B300 die.

DRAM offers roughly three orders of magnitude more storage density than on-die SRAM.

“it's possibly multiple orders of magnitude, three orders of magnitude difference in density for DRAM versus SRAMM”

quantity · guest

plausible · medium confidence — His own numbers give 288GB versus ~0.5GB, i.e. ~2.8 orders of magnitude of packaged capacity — but that conflates cell density with 3D stacking across many HBM layers. On pure cell area, DRAM (~6F²) versus 6T SRAM (~100–150F²) is closer to one to two orders of magnitude. 'Three orders' is right for what a package holds, loose for what 'density' usually means.

To check: Published SRAM bit-cell area at N3/N5 versus DRAM cell area at 1α/1β nodes, and HBM3E stack layer counts.

Cerebras quotes 21 petabytes per second of SRAM bandwidth against roughly 10 terabytes per second for Blackwell's HBM.

“Serbis quits pabytes per second, 21 pabytes per second further away for scale engine 3”

quantity · guest

consistent · high confidence — Cerebras publishes 21 PB/s aggregate on-wafer memory bandwidth and 44GB SRAM for WSE-3 (he says ~50GB later, close enough). Blackwell HBM3E bandwidth is 8 TB/s per GPU, so his '10 terabytes per second or so' is right. The comparison is apples-to-oranges in the way vendors always make it — aggregate wafer-wide versus per-package — but the numbers themselves are as quoted.

To check: Cerebras WSE-3 datasheet (21 PB/s, 44GB SRAM) versus NVIDIA B200/GB300 HBM bandwidth specs.

KV cache memory routinely exceeds the size of the model weights themselves.

“it frequently gets to be larger than the weights of the model themselves”

assertion · guest

consistent · medium confidence — Holds readily at serving scale: a GQA model like Llama-3-70B costs roughly 300KB of KV per token, so a few hundred thousand tokens in flight across a batch exceeds the 140GB of weights. It is much less true for MLA architectures (DeepSeek-V2/V3 cut KV by ~90%+) or short-context single-stream use, hence 'frequently' rather than 'always' is the right word.

To check: KV-cache-bytes-per-token arithmetic for a given model config versus weight size, or vLLM memory profiler output at production batch sizes.

The guest credits Anthropic with being first to a one-million-token context window, while noting he compacts long before hitting it.

“Enthropic was I think the first to hit the 1 million context window length”

assertion · guest

inaccurate · high confidence — Google shipped a 1M-token context window with Gemini 1.5 Pro in February 2024 (with 10M demonstrated in the research report); Anthropic reached 1M for Sonnet 4 only in August 2025. Magic.dev also announced a 100M-token research model in 2024. He hedges with 'I think', which is the honest marking, but the fact is wrong.

To check: Gemini 1.5 technical report (Feb 2024) versus Anthropic's 1M-context announcement date.

The guest predicts SRAM-heavy chipmakers like Cerebras and Groq end up as accelerators paired with GPUs rather than standalone serving platforms.

“Crisis and Grock and maybe a couple others, you should think of them as accelerators”

prediction · guest

plausible · medium confidence — The premise is real — pure-SRAM machines have a hard capacity ceiling on a dynamically growing KV cache, which is why Groq needs many racks per model. Heterogeneous serving and prefill/decode disaggregation are active directions. But these firms have viable standalone businesses in latency-critical niches, and 'accelerator paired with a GPU' is one plausible outcome among several, not a demonstrated trajectory.

To check: Announced products or partnerships pairing Cerebras/Groq silicon with GPU-hosted attention in a single serving path, and their revenue mix over the next few years.

Transformers pair a memory-bound attention layer directly with a compute-bound MLP, which no single chip is good at serving.

“the original sin of transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a computebound layer”

contrarian · guest

consistent · high confidence — The mechanism is standard roofline analysis: attention decode is memory-bandwidth-bound on KV reads, MLP is compute-bound at large batch, and no single memory hierarchy is optimal for both. This is precisely the reasoning behind prefill/decode disaggregation and behind MLA/sparse-attention work. The 'original sin' phrasing is his own and is a nice compression of a known result.

To check: Arithmetic-intensity measurements for attention versus MLP kernels at long context and large batch, e.g. from FlashAttention or DistServe papers.

The efficient future architecture splits attention onto GPUs and the MLP onto SRAM-rich accelerators.

“you would like to put the uh attention possibly on the GPU and the MLP on the cerebrus chip”

prediction · guest

plausible · low confidence — The architectural logic follows from claim 23 and echoes real disaggregation research. The tag-on assertion that 'this is what's happening with Nvidia and Groq' I cannot confirm at all — I know of no such split-execution product, and cross-vendor interconnect latency would be the obvious blocker.

To check: Any shipping system that runs MLP and attention layers on different vendors' silicon within one forward pass, plus its measured interconnect overhead.

Transformers' significance is that they scaled: they carried model sizes from the millions typical of computer vision to trillions.

“parameters and transformers are the link to go from millions to trillions of parameters”

assertion · guest

consistent · high confidence — The numbers check out: AlexNet ~60M, VGG-16 ~138M, so '150M was a huge computer-vision model' is right, and today's MoEs are at 671B (DeepSeek-V3) to 1T+ (Kimi K2). Attributing the scaling to transformers' lack of architectural priors plus their hardware-friendliness is the standard account.

To check: Parameter counts across the CV-to-LLM transition, e.g. Epoch AI's model database.

The internet supplied a one-time data endowment of roughly 30 trillion high-quality tokens (300 trillion on a looser definition), which models have already consumed.

“extremely high quality about 30 trillion tokens of high quality text. Uh 300 trillion tokens if you take a wider view”

quantity · guest

consistent · medium confidence — These match the commonly cited figures: RedPajama-v2 at ~30T tokens, FineWeb ~15T, and the Villalobos/Epoch 'Will we run out of data?' estimate of roughly 300T tokens of public human text (with wide error bars, 100T–3,000T). 'Models have seen it many times over' is an overstatement — no single training run uses all of it — but the endowment sizing is right.

To check: Epoch AI's public human text stock estimate and the token counts of RedPajama-v2 / FineWeb / DCLM.

Feedback from ordinary users no longer improves models because the median served model already exceeds the median human rater.

“unconditioned human preference is not actually worth anything anymore. You want expert human preference at this point”

contrarian · guest

contested · medium confidence — The directional claim is widely held — labs have shifted to expert/domain annotators and the scalable-oversight literature is premised on raters being outmatched. But 'not worth anything' is too strong: aggregated preference signal still trains reward models effectively, and Chatbot Arena-style crowd preference remains a real (if criticised) signal. The disagreement is between RLHF practitioners and the RLVR camp, and it is live.

To check: Ablations comparing reward models trained on crowd versus expert preference data at current model capability levels.

AGI is reached by accumulating specialised, environment-trained intelligences until no capability gaps remain.

“want artificial general intelligence the best way to get there is to just keep stacking specialized intelligences until you have no more gaps to fill”

assertion · guest

contested · medium confidence — This is a real position in the field (the 'RL environments all the way down' view held by e.g. Mechanize and parts of the RLVR community), but it is directly opposed by the scale-and-generalisation camp, who argue capability transfer across domains — not enumeration of domains — is what has produced every jump so far. Whether narrow RL training generalises out of distribution is an open empirical question.

To check: Transfer studies measuring whether RL training on domain A improves held-out performance on unrelated domain B, at frontier scale.

Verifiability plus self-grading is the necessary and sufficient condition for recursive model self-improvement on a task.

“you must make sure that your task is verifiable. You need to give the model a self-grading system. If you have that, you have the recipe for self-improvement on any task you like.”

assertion · guest

contested · high confidence — Verifiability is widely accepted as close to necessary for RLVR, and this is roughly Jason Wei's 'verifier's law'. But sufficiency is where it breaks: reward hacking against imperfect verifiers is thoroughly documented, exploration in sparse-reward environments frequently stalls, and self-grading is precisely the case where the verifier can be gamed. 'Recipe for self-improvement on any task you like' overstates a real principle.

To check: Documented RLVR failures from reward hacking (e.g. specification-gaming case studies) versus successful cross-domain RLVR results on non-code, non-math tasks.

Frontier labs have shifted budget from data acquisition to RL environments.

“labs spend. They used to spend much that much on data. Now they spend a lot more on RL environments.”

assertion · guest

plausible · medium confidence — Consistent with everything visible from outside: Scale AI's disruption and Surge/Mercor's growth, purpose-built environment startups (Prime Intellect, Mechanize), and reported nine-figure lab budgets for environments during 2025. He concedes it is not a differentiated take. Exact spend ratios are not public.

To check: Disclosed contract values or revenue figures from environment and data vendors, or lab spend breakdowns in funding-round reporting.

Kernel engineering has moved from hand-writing code to specifying strategy in natural language and letting models implement it.

“he taught me was uh you shouldn't write kernels by hand anymore necessarily”

assertion · guest

plausible · medium confidence — Tri Dao is a real and load-bearing citation here (FlashAttention, Together AI chief scientist), and model-assisted kernel authoring is genuinely how many teams now work — KernelBench exists precisely to measure it. The caveat is that frontier-competitive kernels still typically require human architectural insight, and at least one widely publicised claim of AI-authored kernel speedups (Sakana's CUDA result) turned out to be a measurement artefact.

To check: KernelBench scores for current models, and whether shipped production kernels in vLLM/SGLang are model-authored.

Software efficiency will stop being a durable edge because frontier model improvements lift every inference provider's kernel engineering equally.

“we're all we're all downstream of the frontier in terms”

assertion · guest

plausible · medium confidence — Coherent and notably candid — it argues against his own moat. The counterweight is that access to the hardware, profiling data and workload mix is not uniformly distributed, so 'lifts all boats equally' is the strong form; teams with better in-context knowledge of their own fleet will still extract more.

To check: Convergence (or not) of tokens/sec/GPU across providers serving the same open model on the same hardware, tracked over time.

On large matrix multiplies GPUs already hit 70-80% of peak, limited by power and thermals rather than software — so the inefficiency lies in time spent outside that path.

“that operation runs at you know 70 80% of peak utilization and it's limited not by software but by power”

quantity · guest

consistent · high confidence — Well-matched to reality: large BF16/FP8 GEMMs via cuBLAS/CUTLASS land in the 70–85% of theoretical peak range, and sustained dense tensor-core work does trigger clock reduction against power and thermal limits, which is why NVIDIA's headline peak flops are not achievable in steady state. His broader point — that the inefficiency lives outside the GEMM — is the right diagnosis.

To check: Nsight Compute or DCGM traces on a large GEMM showing achieved flops versus spec peak, alongside SM clock and power draw.

Nvidia deliberately rations chips rather than clearing the market on price, because concentrating compute in a few customers would empower them.

“happier but Nvidia sees the if they just let the most deep pockets buy all the chips that maybe hurts them in the long term”

assertion · guest

plausible · medium confidence — The observable behaviour — allocation by relationship rather than pure price-clearing, and deliberate cultivation of neoclouds — is well documented. The motive he attributes (avoiding customer concentration of power) is inference, not evidence, and competing explanations exist: long-term demand smoothing, avoiding a price-spike-then-crash cycle, and antitrust optics.

To check: Nvidia's disclosed customer concentration in 10-K filings, and any internal or on-record account of allocation criteria.

No accelerator is inherently bad; every chip has a comparative advantage that the right price makes exploitable.

“inferior. I I like to say there's no bad chips. There's really bad pricing. And uh I will make any chip work at the right price.”

assertion · guest

plausible · medium confidence — A trader's framing of a real fact: flops-per-dollar leadership does rotate away from NVIDIA on second-tier silicon. The elided cost is engineering — porting kernels and serving stacks across TPU, Trainium, AMD and startup ISAs has a fixed cost that only amortises at volume, which is exactly why most competitors don't do it.

To check: Achieved tokens/sec/dollar on non-NVIDIA silicon in their fleet versus the same workload on H100/B200.

The market's underrating of AMD chips creates arbitrage for teams that can write good kernels for them, though the guest concedes this is closing.

“people have this perception that AMD is not as as good as Nvidia. That's music to my ears”

assertion · guest

consistent · high confidence — The supporting facts are right: OpenAI signed a multi-gigawatt MI450 deal with AMD in October 2025 including a large warrant, and Meta has run production inference on MI300X. His own concession that the arbitrage is closing is the accurate part; MI300X memory capacity has been a genuine inference advantage while ROCm software maturity has been the genuine gap.

To check: AMD data-center GPU revenue disclosures and the OpenAI/Meta deal terms in public filings.

Unlike dot-com era networking buildout, token demand is realised immediately rather than speculatively anticipated.

“longer speculative. People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately.”

assertion · guest

contested · medium confidence — Trivially true at the token level — tokens are non-storable, unlike dark fibre. But it does not carry the weight he puts on it: the capital is spent on the datacenters and chips that produce the tokens, and that capacity is absolutely built ahead of demand. Bears (and several 2025-26 analyses of vendor-financing and circular deals) argue exactly that the buildout, not the token purchase, is the speculative object.

To check: Utilisation rates of newly commissioned AI capacity, and the ratio of announced capex to realised inference revenue across hyperscalers.

The guest predicts inference spending only rises, because it is demand-driven rather than speculative.

“I do think inference spend monotonically increases. Uh there's no speculation on inference spend.”

prediction · guest

unverifiable · medium confidence — A forecast with an unusually strong word — 'monotonically' admits no quarter of decline. The Jevons argument behind it is reasonable and has held so far, but falling per-token prices could in principle depress aggregate spend, and his own claim 60 about needing 3–6 orders of magnitude of cost reduction sits in tension with unbroken spend growth.

To check: Quarterly inference revenue across major providers over several years; any single quarterly decline falsifies the strong form.

Training clusters can always serve inference but not the reverse, because training demands far more inter-chip networking.

“and training is the superset workload over inference you can make any training cluster work for inference but maybe not vice versa”

assertion · guest

consistent · medium confidence — Correct on the main axis: training demands scale-out bandwidth (InfiniBand/RoCE fabrics, large coherent domains) that inference-only fleets skip. Two caveats — very large MoE inference with expert parallelism is closing that gap and now wants big NVLink domains too, and some inference-specialised silicon (Groq, older inference accelerators) genuinely cannot train at all, which is the 'not vice versa' half.

To check: Network bandwidth-per-GPU specifications of training versus inference-optimised cluster designs, and whether inference-first fleets are ever repurposed for frontier training.

Concentrated power at gigawatt scale is effectively unobtainable in the US, and even 100MW sites are increasingly hard to secure.

“basically there's no way to build a gigawatt data center in the United States easily anymore”

assertion · guest

contested · medium confidence — Gigawatt-class campuses are demonstrably being built — Stargate Abilene, xAI's Memphis site, Meta's Hyperion and Prometheus — so the literal claim is wrong, but he qualifies with 'easily' and 'unless you're a very special set of customers', which is fair: multi-year interconnection queues, transformer lead times and local opposition make it accessible only to the deepest-pocketed buyers. The strong reading conflicts with observable builds; the hedged reading is right.

To check: Interconnection queue times by ISO/RTO, and the count of >1GW AI campuses that reached energisation in 2025–26.

10MW sites sit at the edge of feasibility while 1MW pockets of power are abundant and ignored by training-oriented buyers.

“megawatts is probably on the edge of what's possible today and 1 megawatt I would argue is plentiful”

quantity · guest

contested · medium confidence — 10MW is a routine colocation footprint, so 'edge of what's possible' reads as overstated even in a tight market; North American colo vacancy has been near record lows and pre-leasing high, which supports scarcity but not that degree. The genuinely interesting half — that ~1MW pockets are abundant and ignored because training buyers need concentration — is a real and under-discussed observation.

To check: CBRE/datacenterHawk North America vacancy and available-capacity-by-size reports, and quoted lead times for 1MW versus 10MW deals.

With liquid cooling, a megawatt of compute now fits into roughly eight refrigerator-sized racks.

“around like eight racks worth of compute. Each rack is about the size of a refrigerator”

quantity · guest

consistent · high confidence — Arithmetic checks: a GB200 NVL72 rack draws roughly 120–132kW, so eight racks is about one megawatt. Rack physical dimensions are indeed roughly a large refrigerator. This is a good example of him giving a number that can be verified rather than gestured at.

To check: NVIDIA GB200/GB300 NVL72 rack power specifications and physical dimensions.

The company claims a flexibility and liquidity in compute purchasing that no competitor has.

“describe what we do is we will buy any chip anywhere in the world for any duration of time”

assertion · guest

unverifiable · high confidence — Company self-description and a competitive positioning claim. Others make similar claims about flexible compute aggregation (SF Compute, various brokers); 'no one else has this' is not checkable from outside.

To check: The actual heterogeneity of their fleet — vendor and generation mix — and the range of contract durations they have transacted.

Because his workloads are asynchronous and his control plane reschedules failures, uptime is a linear price trade rather than a cliff.

“happen at some rate and I am basically linearly happy with a data center that's 95% uptime versus 98% versus 99%”

contrarian · guest

plausible · medium confidence — Internally coherent given async workloads and a rescheduling control plane, and it is a genuinely differentiated position — 95% uptime means roughly 18 days a year of downtime, which is disqualifying for conventional buyers. The linearity claim depends on failures being uncorrelated across his sites, which he states as an assumption without evidence; correlated failure (grid events, weather systems) is exactly the case that breaks it.

To check: Measured correlation of outage events across their sites, and delivered tokens-per-dollar at 95% versus 99% sites.

The company explicitly sells uncontrolled tail latency in exchange for the lowest price — the right trade for background agents.

“be very competitive, but our P99, our 99th percentile latency, it's not going to be controlled. It cannot be. And in return, I'll give you unbeatable economics.”

assertion · guest

unverifiable · high confidence — A report of what he tells customers — private commercial information. The trade itself is a clean, unusual piece of product design: explicitly selling away tail latency is the mirror image of every SLA in the industry, and it only works if the long-horizon-agent thesis (claim 4) holds.

To check: Their published SLA terms, and third-party latency measurements showing median-competitive but heavy-tailed distributions.

Tolerating multi-day outages lets him use intermittent solar and wind power that no other data centre buyer will touch.

“totally capable of tolerating a outage for my data center that's measured in in even days or weeks”

assertion · guest

plausible · medium confidence — Curtailment-tolerant and demand-flexible compute is a real and growing idea (ERCOT demand response, Emerald AI, flexible-load interconnection proposals), and weather is indeed forecastable at the relevant horizon. The unexamined economics: with GPUs as the dominant capex, an 80% capacity factor raises cost per delivered token by ~25%, so this only pencils if the chips are cheap enough — which he asserts but does not quantify.

To check: A cost model comparing $/token at 80% versus 99% capacity factor, given their actual chip capex and PPA prices.

The strategy is deliberately non-competitive with the frontier labs for compute, targeting supply they cannot see as usable.

“The idea is in both cases I do not want to be in bidding against Anthropic or Open AI for compute capacity”

assertion · guest

unverifiable · high confidence — A strategy statement, and a candid one — he concedes outright he would lose the bidding war. The 'compete where incumbents won't look' framing is classic disruption positioning; whether the neglected supply is large enough to matter is the open question, not the intent.

To check: The realised size and cost basis of their fleet relative to frontier-lab capacity over time.

Accumulated small, distributed pools of compute will eventually form a token factory with unbeatable unit economics.

“to get concentrated supply. I will only get aggregate supply. And over time I build my aggregate factory that is unbeatable in economics.”

prediction · guest

unverifiable · medium confidence — A forecast about his own business with no numbers attached. The unaddressed costs of the distributed strategy are real: per-site fixed overhead, remote hands, networking, and — most substantively — that trillion-parameter MoE serving increasingly wants large coherent NVLink domains, which is exactly what a fleet of 1MW sites cannot provide. The mini-mill analogy is apt but mini-mills won on scrap feedstock economics, not on coordination.

To check: Their published price per million tokens versus hyperscaler and neocloud competitors at equivalent model and throughput tier.

Modern mixture-of-experts models activate under 10% of experts and frontier models nearer 1%, so compute is already used judiciously.

“are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated. And I think the frontier models are closer to like 1%.”

quantity · guest

consistent · medium confidence — The open-model figures support him: DeepSeek-V3 activates 8 of 256 routed experts (37B of 671B active, ~5.5%), Kimi K2 ~32B of 1T (~3%), Llama 4 Maverick 17B of 400B, Qwen3-235B-A22B ~9%. The '~1% at the frontier' half is explicitly his estimate about undisclosed architectures and should be treated as such; earlier-generation MoEs like Mixtral were far denser at 25% of experts.

To check: Published active/total parameter ratios and expert-activation counts in model cards for DeepSeek, Kimi, Qwen, Llama 4; frontier figures would need a leak or disclosure.

KV cache storage is roughly one to two orders of magnitude larger than the entropy it carries, making it the biggest remaining inefficiency.

“many kilobytes of data in the KV cache per token. Um, and that's probably off by an order of magnitude or two.”

quantity · guest

plausible · medium confidence — The supporting evidence is real: DeepSeek's MLA cut KV cache by roughly 93% versus MHA, and their sparse-attention work continued that, so order-of-magnitude annual progress is a fair characterisation. The 'entropy' argument is a heuristic rather than a measurement — nobody has established the information-theoretic floor for KV representations — so 'off by one to two orders' is an educated guess he correctly marks as such.

To check: KV-bytes-per-token across MHA/GQA/MLA/sparse variants at matched quality, plus any published lower-bound analysis of KV compressibility.

Nvidia will ship roughly 5 million Blackwell chips this year, yet much of that fleet sits idle.

“all this compute in the world. Nvidia is pumping out 5 million Blackwell chips this year”

quantity · guest

plausible · medium confidence — In the right ballpark — NVIDIA and analysts discussed Blackwell shipments in the several-million-unit range for its first year, and CoWoS capacity expansion supports that order. I can't pin the specific year he means given the Rubin transition, and 'chips' versus 'GPU dies' versus 'packages' materially changes the count.

To check: NVIDIA quarterly data-center revenue and unit disclosures, or TSMC CoWoS allocation estimates from TrendForce/Morgan Stanley.

Global compute is badly orchestrated; large numbers of GPUs sit unused in private pools, worse than XAI's much-mocked utilisation.

“it's far far worse. A ton of GPUs just sit in warehouses or sit in private pools allocated to a specific customer um just don't get utilized.”

assertion · guest

unverifiable · medium confidence — No public dataset exists on global GPU utilisation, and he flags his own uncertainty ('I certainly doubt it'). Directionally believable — reserved-but-unused capacity is a known feature of multi-year contracts, and the existence of spot compute marketplaces is itself evidence of stranded capacity — but 'far far worse' has no number behind it, and he has an obvious commercial interest in the claim.

To check: Aggregate utilisation telemetry from neoclouds or a spot-market clearing-price series indicating persistent excess supply.

He hires for curiosity and love of performance engineering, treating CUDA or AI experience as a red herring.

“Curiosity. It's 100% curiosity. You know the one thing I cannot teach is love for performance”

assertion · guest

unverifiable · high confidence — Hiring philosophy — private and unfalsifiable. The supporting argument (CUDA-specific experience decays because the programming model has changed substantially across Volta→Hopper→Blackwell, warp specialisation, TMA, cluster-level programming) is technically sound and a non-practitioner would not make it.

To check: Nothing external; at most, the actual background composition of their engineering hires.

Frontier labs spend enormously to hold a three-to-six-month capability lead, which he thinks is still worth it today.

“in a line. I would say the labs pay an immense premium to be 3 to 6 months ahead of of everything else”

assertion · guest

plausible · medium confidence — The 3–6 month closed-to-open gap is the commonly cited figure and matches observed cases (DeepSeek-R1 landing roughly four months after o1; open reasoning models trailing frontier releases by two quarters). Whether the premium paid is 'immense' relative to that lead is a judgement, and he immediately doubts its durability, which is the intellectually honest move.

To check: Benchmark-parity dates between frontier closed models and the first open model matching them, tracked across several release cycles.

The guest would not be surprised if a frontier-class coding model could be trained purely on AI-generated open-source code already public on GitHub.

“all we need. I would not be surprised if you could train a fable glass model only on the outputs of code you consider good on GitHub that's open source.”

prediction · guest

contested · medium confidence — Properly hedged, and the 'latent distillation' mechanism is real — a large and rising share of public code is model-generated, and distillation from outputs demonstrably transfers capability (Alpaca through DeepSeek's distilled models). Against it: the model-collapse literature (Shumailov et al.) on recursive training on synthetic data, and the fact that frontier coding capability now comes substantially from RL against execution feedback, not from imitating static code.

To check: A controlled experiment training a coding model exclusively on filtered AI-generated public repos and evaluating on SWE-bench Verified against a frontier baseline.

Preventing capability diffusion from closed to open models is impossible; only the speed is in question.

“I don't think it's fundamentally possible to prevent the diffusion of of information or model capabilities. It will happen. The question is just how fast.”

assertion · guest

plausible · medium confidence — Consistent with the historical record on technology diffusion and with the observed 3–12 month replication lag for each frontier capability. It is a stated impossibility claim, which cannot be proved, and export controls plus weight-security programmes are precisely the attempt to test it. The honest framing — only speed is in question — is what most informed people would agree with.

To check: Replication lag between each frontier capability's first demonstration and its first open-weights equivalent, tracked over successive generations.

Enterprise adoption lags the frontier by more than the labs' three-to-six-month lead, undercutting the value of that lead.

“A lot of enterprises are probably still on like 46, Opus 46 or Opus 47. They don't they don't adopt the bleeding edge rapidly.”

assertion · guest

plausible · medium confidence — The general pattern is well established — enterprises pin model versions to avoid re-running evals, re-tuning prompts and re-certifying compliance, and deprecation notices are a standard source of enterprise complaint. The specific model versions named are past my knowledge, and 'probably' marks it as his impression rather than data.

To check: Model-version distribution in enterprise API traffic, e.g. from Anthropic/OpenAI enterprise usage reports or third-party gateway telemetry.

Open-source model development is structurally permanent because incentives guarantee replacement leaders.

“and fundamentally I don't think open source ever goes away. If there's a vacuum because one leader steps out, a new leader will step in.”

assertion · guest

plausible · medium confidence — Well supported by the recent record: as Meta's Llama leadership faded through 2025, DeepSeek, Qwen, Kimi, GLM and Mistral filled the gap, and the sovereignty motive he cites earlier is a real driver of national and enterprise demand. The structural argument (incentives guarantee replacement) is an assertion, but the empirical pattern currently backs it.

To check: Continuity of frontier-adjacent open-weight releases and their download/deployment share over successive years.

The difficulty of training a frontier-class model declines continuously.

“easier every day to treat to train a frontier class model”

assertion · guest

contested · medium confidence — Depends entirely on reading. Cost to reach a *fixed* capability level falls very fast — algorithmic efficiency plus hardware gains compound to roughly an order of magnitude per year by Epoch's estimates. But cost to reach the *moving frontier* has risen sharply: frontier training compute grows ~4–5x annually and top runs are in the hundreds of millions to billions of dollars. He says 'frontier class', which is the harder reading, and offers no evidence.

To check: Epoch AI's series on frontier training-run cost over time versus cost-to-reach-fixed-benchmark over time.

The guest's desired end state is cheap abundant tokens with every company and user owning a customised harness, mostly via in-context learning rather than fine-tuning.

“I want abundant tokens and diverse harnesses”

assertion · guest

unverifiable · high confidence — A stated preference for the future, not a forecast or fact. The technical sub-claim — that customisation goes through in-context learning rather than weight fine-tuning — is a real and currently ascendant view (context engineering, agent memory systems, long-context retrieval), though the counter-camp points to serving-cost advantages of small fine-tuned models and to RFT offerings from the labs.

To check: Relative adoption of fine-tuning APIs versus context/memory-based customisation in production agent deployments.

A trillion tokens costs at least $5 million at current OpenAI frontier pricing.

“a trillion tokens well okay at openi pricing that's at least $5 million at the very least for 5.5 or 5.6”

quantity · guest

contested · medium confidence — The arithmetic looks high as a floor. At GPT-5-class published pricing (~$1.25/M input, $10/M output), a trillion mostly-input tokens is about $1.25M and a realistic mixed workload lands nearer $2–3M; only output-heavy mixes reach $10M. So '$5 million at the very least' is a blended midpoint presented as a lower bound. Pricing for the versions he names is past my knowledge, which is the main caveat.

To check: Published per-million-token input/output prices for the named models, multiplied against a stated input/output ratio.

Making trillion-token-per-day consumption ordinary requires three to six orders of magnitude of cost-per-token improvement.

“least at least um you know, three to six orders of magnitude improvement in cost per token”

quantity · guest

consistent · medium confidence — Internally consistent arithmetic given his $5M-per-trillion premise and a $5,000-to-$5 target range. Worth noting against it that cost per unit of capability has already fallen dramatically — a16z's 'LLMflation' analysis put it near 1,000x over three years for fixed MMLU level — so three orders is aggressive but not fantastical; six orders would require far more than serving efficiency.

To check: A price-per-token-at-fixed-capability time series (Epoch or Artificial Analysis) extrapolated forward.

Demand for intelligence is never the binding constraint; the constraint is the on-ramps product people build.

“I never will believe in that. There is always demand for intelligence in the world.”

assertion · guest

unverifiable · high confidence — Explicitly framed as an article of faith ('I never will believe'), which puts it outside evidence. The empirical question underneath — the price elasticity of demand for inference, and whether Jevons holds indefinitely — is genuinely open, and his only support is anecdotal customer reports of rationing free-tier tokens.

To check: Measured elasticity: token consumption response to large price cuts on comparable models, across several such cuts.

Like-for-like performance per watt has barely improved from Hopper to Blackwell to Rubin, and similarly across TSMC nodes.

“performance per watt of Bloat 16 multiply it hasn't improved all that much”

contrarian · guest

plausible · medium confidence — Directionally defensible and a useful corrective. Roughly: H100 SXM at ~990 BF16 dense TFLOPS / 700W ≈ 1.4 TFLOPS/W; B200 at ~2,250 / 1,000W ≈ 2.25 TFLOPS/W — about 1.6x per generation, far below the headline 30x figures that come from FP4, NVLink scale-up and software. So 'not dramatic' is fair, though 1.6x per generation compounded is not nothing. On nodes, TSMC's own claims of ~25–30% power reduction per step from N5 to N3 to N2 support him per-step while summing to ~2x cumulatively.

To check: Spec-sheet BF16 dense TFLOPS and TDP for H100/H200/B200/GB300, and TSMC's published node-to-node power/performance claims.

Losing access to TSMC would be a supply shock but not catastrophic, because western process nodes are closer than chip discourse suggests.

“my contrarian take is that it wouldn't be that bad”

contrarian · guest

contested · medium confidence · novel — A genuinely contrarian framing, and the perf-per-watt premise partly supports it. But it underweights the binding constraint most analysts name: not node quality but volume and advanced packaging. TSMC holds the overwhelming majority of leading-edge foundry capacity and essentially all high-volume CoWoS; Intel Foundry's external leading-edge volume is a small fraction of that. A TSMC loss would be a capacity catastrophe long before it was a perf-per-watt problem, and he does not engage that.

To check: Leading-edge wafer capacity and CoWoS/advanced-packaging capacity by supplier, from TrendForce or company disclosures.

Intel's best process is at worst about 2x behind TSMC on performance per watt.

“west uh like Intel not that far behind at worst like maybe 2x uh worse performance per watt”

quantity · guest

plausible · low confidence — Offered explicitly as a bound ('at worst'), and it looks generous — independent assessments generally place Intel 18A somewhere between TSMC N3 and N2 on power/performance, i.e. well inside 2x. The honest caveat is that no high-volume AI accelerator has shipped on Intel's leading edge, so the comparison is inferred from CPU products rather than measured on the relevant workload.

To check: Measured perf/watt of a comparable design fabricated on Intel 18A versus TSMC N3/N2 — Clearwater Forest and Panther Lake data are the closest available proxies.

HBM is the supply bottleneck most worth attacking because memory fab capacity cannot be expanded quickly.

“Why? There's no easy way to bring on a lot more fabs of memory”

assertion · guest

consistent · medium confidence — Matches the 2025–26 picture: HBM capacity was effectively sold out well in advance at SK Hynix and Samsung, conventional DRAM was pulled into shortage as capacity shifted to HBM, and the memory makers' capex conservatism after repeated cycle burns is a documented industry pattern. New DRAM fab lead times run several years. The 'boys in Boise' line about Micron is apt.

To check: Micron/SK Hynix/Samsung capex guidance and announced HBM capacity additions versus stated sold-out horizons; DRAM contract price indices.

The memory shortage will raise prices across consumer electronics rather than being routed around by efficiency gains.

“I think they're gonna make everything else more expensive. Think that iPhones will cut their memory. iPhones are going to go up in price”

prediction · guest

plausible · medium confidence — The mechanism is real and was already visible by late 2025 — DRAM and NAND contract prices rose sharply and several phone makers publicly warned of cost pressure and price increases in 2026. The specific prediction that Apple in particular cuts memory configurations is speculation; Apple's supply contracts and margins give it more insulation than most. He offers no evidence, just a direct rebuttal to the host.

To check: TrendForce DRAM/NAND contract price series, and announced 2026 smartphone BOM/pricing changes including Apple's configuration tiers.

Nvidia deliberately refrains from selling tokens or running clouds so as to keep a competitive, dependent ecosystem of buyers.

“Nvidia is really smart about this? They don't compete with their customers.”

assertion · guest

contested · medium confidence — The broad strategy is real — NVIDIA cultivates neoclouds, and its CoreWeave stake and capacity backstops are documented. But 'doesn't compete with customers' is too clean: DGX Cloud is a first-party cloud offering that irritated partners, NIM microservices sell inference software, and NVIDIA has taken equity positions across the buyer base. The framing serves his own narrative about ecosystem-friendly supply.

To check: NVIDIA's DGX Cloud revenue and positioning in its 10-K, and its disclosed equity stakes in cloud and inference providers.