Jeff Dean and Bill Dally on Why Inference Is Eating AI
At GTC 2026, two industry leaders explained why running AI models now matters more than training them—and what hardware needs to change.
@JeffDeanTL;DR:
- Inference now uses up to 90% of data center power, dwarfing training workloads
- The bottleneck isn't computation—it's moving data around the chip
- Target speed for AI agents: 10,000-20,000 tokens per second per user
- Future training may shift from static datasets to agents learning from their environment
Headline
Jeff Dean and Bill Dally Think Inference Is the Real Challenge Now
Summary
At NVIDIA's GTC 2026, Jeff Dean (Google DeepMind) and Bill Dally (NVIDIA) talked through a shift that's been building for years: inference has become the dominant AI workload. They estimate inference now eats up to 90% of data center power. The problem isn't raw compute—it's getting data where it needs to go fast enough.
Their proposed fixes include redesigning on-chip communication, building custom interconnects, and using different hardware for different parts of the inference pipeline. The goal is ambitious: 10,000-20,000 tokens per second per user, which would let AI agents run autonomously for hours or days on complex tasks.
This matters because scaling AI agents requires hardware that doesn't exist yet. Dean also floated a different approach to training—instead of feeding models static data, let them learn by interacting with environments.
Analysis
The focus on inference makes sense when you look at where AI is actually being used. Training happens once (or occasionally); inference happens constantly, across millions of users. Agent-based systems make this worse because they need responses fast enough to act autonomously.
Multiple sources—including NVIDIA session recaps and technical analyses—point to data movement as the core bottleneck. Fixes like tile-based designs and simplified PHY interfaces keep coming up. This creates an interesting competitive dynamic: NVIDIA's GPUs versus Google's TPUs, both racing to build better inference hardware.
Dean's comments about "active" training are worth watching. Instead of scraping the internet for more data, future models might learn by doing—interacting with simulations or the real world, generating their own training signal. This could help with the looming data scarcity problem.
One practical note: NVIDIA already uses LLMs (like Chip NeMo) to help design chips. The tools are building their own future hardware.
Impact Assessment
- Significance: High
- Categories: Technical Insight, AI Research, Industry Trend