TPUs vs. GPUs and why Google is positioned to win AI race in the long term
- Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects.
To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale Nvidia system based on 144 “Blackwell” GPU chiplets with an aggregate of 20.7 TB of HBM memory look like a joke."
Nvidia may have the superior architecture at the single-chip level, but for large-scale distributed training (and inference) they currently have nothing that rivals Google's optical switching scalability.
- I always enjoy being wrong and I was very wrong in my predictions about Google : I thought they should theoretically win, but I was also very confident they couldn't possibly turn their execution ship around to actually pull together a coherent competitor to OpenAI. But they do seem to have done that and it's very impressive. If they do continue to execute, I can't see anybody stopping them dominating and I would be bearish on nearly every other player catching them.
The biggest problem though is trust, and I'm still holding back from letting anyone under my authority in my org use Gemini because of the lack of any clear or reasonable statement or guidelines on how they use your data. I think it won't matter in the end if they execute their way to domination - but it's going to give everyone else a chance at least for a while.
by lukeschlather
3 subcomments
- This feels a lot like the RISC/CISC debate. More academic than it seems. Nvidia is designing their GPUs primarily to do exactly the same tasks TPUs are doing right now. Even within Google it's probably hard to tell whether or not it matters on a 5-year timeframe. It certainly gives Google an edge on some things, but in the fullness of time "GPUs" like the H100 are primarily used for running tensor models and they're going to have hardware that is ruthlessly optimized for that purpose.
And outside of Google this is a very academic debate. Any efficiency gains over GPUs will primarily turn into profit for Google rather than benefit for me as a developer or user of AI systems. Since Google doesn't sell TPUs, they are extremely well-positioned to ensure no one else can profit from any advantages created by TPUs.
- This is highly relevant:
"Meta in talks to spend billions on Google's chips, The Information reports"
https://www.reuters.com/business/meta-talks-spend-billions-g...
by bastawhiz
7 subcomments
- I don't think what the article writes about matters all that much. Gemini 3 Pro is arguably not even the best model anymore, and it's _weeks_ old, and Google has far more resources than Anthropic does. If the hardware actually was the secret sauce, Google would be wiping the floor with little everyone else.
But they're not.
There's a few confounding problems:
1. Actually using that hardware effectively isn't easy. It's not as simple as jacking up some constant values and reaping the benefits. Actually using the hardware is hard, and by the time you've optimized for it, you're already working on the next model.
2. This is a problem that, if you're not Google, you can just spend your way out of. A model doesn't take a petabyte of memory to train or run. Regular old H100s still mostly work fine. Faster models are nice, but Gemini 3 Pro being 50% of the latency as Opus 4.5 or GPT 5.1 doesn't add enough value to matter to really anyone.
3. There's still a lot of clever tricks that work as low hanging fruit to improve almost everything about ML models. You can make stuff remarkably good with novel research without building your own chips.
4. A surprising amount of ML model development is boots on the ground work. Doing evals. Curating datasets. Tweaking system prompts. Having your own Dyson sphere doesn't obviate a lot of the typing and staring at a screen that necessarily has to be done to make a model half decent.
5. Fancy bespoke hardware means fancy bespoke failure modes. You can search stack overflow for CUDA problems, you can't just Bing your way to victory when your fancy TPU cluster isn't doing the thing you want it to do.
by 1980phipsi
9 subcomments
- > It is also important to note that, until recently, the GenAI industry’s focus has largely been on training workloads. In training workloads, CUDA is very important, but when it comes to inference, even reasoning inference, CUDA is not that important, so the chances of expanding the TPU footprint in inference are much higher than those in training (although TPUs do really well in training as well – Gemini 3 the prime example).
Does anyone have a sense of why CUDA is more important for training than inference?
- A question I don't see addressed in all these articles: what prevents Nvidia from doing the same thing and iterating on their more general-purpose GPU towards a more focused TPU-like chip as well, if that turns out to be what the market really wants.
- If Google won, it would cannibalize its current ad-driven business and replace it with something that is extremely expensive to run and difficult to make profit from. A Pyrrhic win essentially.
- The funniest thing about this story is that NVIDIA has essentially become a TPU company. Look at the Hopper and Blackwell architectures: Tensor Cores are taking up more space, the Transformer Engine has appeared, and NVLink has started to look like a supercomputer interconnect. Jensen Huang isn't stupid. He saw the threat of specialized ASICs and just built the ASIC inside the GPU. Now we have a GPU that is 80% matrix multiplier but still keeps CUDA compatibility. Google tried to kill the GPU, but instead forced the GPU to mutate into a TPU
- I have read in the past that ASICs for LLMs are not as simple a solution compared to cryptocurrency. In order to design and build the ASIC you need to commit to a specific architecture: a hashing algorithm for a cryptocurrency is fixed but the LLMs are always changing.
Am I misunderstanding "TPU" in the context of the article?
- The part that surprised me is how much TPUs gain from the systolic array design. It basically cuts down the constant memory shuffling that GPUs have to do, so more of the chip’s time is spent actually computing.
The downside is the same thing that makes them fast: they’re very specialized. If your code already fits the TPU stack (JAX/TensorFlow), you get great performance per dollar. If not, the ecosystem gap and fear of lock-in make GPUs the safer default.
- It's all about the right mind set at the very top level. At the beginning of the PC era, nobody would bet IBM to lose. Same in the dawn of internet, all money was on MS. so it happened to Nokia and Ericsson.
Google is a giant without a direction. The ads money is so good that it just doesn't have the gut to leave it on the table.
- Given the importance of scale for this particular product, any company placing itself on "just" one layer of the whole story is at a heavy disadvantage, I guess. I'd rather have a winning google than openai or meta anyway.
- 5 days ago: https://news.ycombinator.com/item?id=45926371
Sparse models have same quality of results but have less coefficients to process, in case described in the link above sixteen (16) times as less.
This means that these models need 8 times less data to store, can be 16 and more times faster and use 16+ times less energy.
TPUs are not all that good in the case of sparse matrices. They can be used to train dense versions, but inference efficiency with sparse matrices may be not all that great.
by thelastgallon
0 subcomment
- With its AI offerings, can Google suck the oxygen out of AWS? AWS grew big because of compute. The AI spend will be far larger than compute. Can Google launch AI/Cloud offerings with free compute bundled? Use our AI, and we'll throw in compute for free.
- It's a cool subject and article and things I only have a general understanding of (considering the place of posting).
What I'm sure about is having a programming unit more purposed to a task is more optimal than a general programming unit designed to accommodate all programming tasks.
More and more of the economics of programming boils down to energy usage and invariably towards physical rules, the efficiency of the process has the benefit of less energy consumed.
As a Layman is makes general sense. Maybe a future where productivity is based closer on energy efficiency rather than monetary gain pushes the economy in better directions.
Cryptocurrency and LLMs seem like they'll play out that story over the next 10 years.
- How much of current GPU and TPU design is based around attn's bandwith hungry design? The article makes it seem like TPUs aren't very flexible so big model architecture changes, like new architectures that don't use attn, may lead to useless chips. That being said, I think it is great that we have some major competing architectures out there. GPUs, TPUs and UMA CPUs are all attacking the ecosystem in different ways which is what we need right now. Diversity in all things is always the right answer.
by sandGorgon
0 subcomment
- deepseek kind of innovated on this using off-the-shelf components right ?
to quote from their paper
"In order to ensure sufficient computational performance for DualPipe, we customize efficient
cross-node all-to-all communication kernels (including dispatching and combining) to conserve
the number of SMs dedicated to communication. The implementation of the kernels is codesigned with the MoE gating algorithm and the network topology of our cluster."
by siliconc0w
4 subcomments
- Google has always had great tech - their problem is the product or the perseverance, conviction, and taste needed to make things people want.
- > The GPUs were designed for graphics [...] However, because they are designed to handle everything from video game textures to scientific simulations, they carry “architectural baggage.” [...] A TPU, on the other hand, strips away all that baggage. It has no hardware for rasterization or texture mapping.
With simulations becoming key to training models doesn't this seem like a huge problem for Google?
by WarOnPrivacy
1 subcomments
- I wish we had more options for a dedicated/stand-alone TPU for end users. I recently bought a 2019 Coral, which as far as I know is my only option.
by clickety_clack
0 subcomment
- Any chance of a bit of support for jax-metal, or incorporating apple silicon support into Jax?
- All this assumes that LLMs are the sole mechanism for AI and will remain so forever: no novel architectures (neither hardware nor software), no progress in AI theory, nothing better than LLMs, simply brute force LLM computation ad infinitum.
Perhaps the assumptions are true. The mere presence of LLMs seems to have lowered the IQ of the Internet drastically, sopping up financial investors and resources that might otherwise be put to better use.
- That and the fact they can self-fund the whole AI venture and don't require outside investment.
- At this stage, it is somewhat clear that it doesn't really matter who's ahead in the race, cause everyone else is super close behind...
- I have never understood why, in these discussions, nobody brings up other specialized silicon providers like Groq, SambaNova, or my personal favorite, Cerebras.
Cerebras CS-3 specs:
• 4 trillion transistors
• 900,000 AI cores
• 125 petaflops of peak AI performance
• 44GB on-chip SRAM
• 5nm TSMC process
• External memory: 1.5TB, 12TB, or 1.2PB
• Trains AI models up to 24 trillion parameters
• Cluster size of up to 2048 CS-3 systems
• Memory B/W of 21 PB/s
• Fabric B/W of 214 Pb/s (~26.75 PB/s)
Comparing GPU to TPU is helpful for showcasing the advantages of the TPU in the same way that comparing CPU to Radeon GPU is helpful for showcasing the advantages of GPU, but everyone knows Radeon GPU's competition isn't CPU, it's Nvidia GPU!
TPU vs GPU is new paradigm vs old paradigm. GPUs aren't going away even after they "lose" the AI inference wars, but the winner isn't necessarily guaranteed to be the new paradigm chip from the most famous company.
Cerebras inference remains the fastest on the market to this day to my knowledge due to the use of massive on-chip SRAM rather than DRAM, and to my knowledge, they remain the only company focused on specialized inference hardware that has enough positive operating revenue to justify the costs from a financial perspective.
I get how valuable and important Google's OCS interconnects are, not just for TPUs or inference, but really as a demonstrated PoC for computing in general. Skipping the E-O-E translation in general is huge and the entire computing hardware industry would stand to benefit from taking notes here, but that alone doesn't automatically crown Google the victor here, does it?
by hirako2000
0 subcomment
- Then Groq should reign emperor?
by DonHopkins
1 subcomments
- Will Google sell TPUs that can be plugged into stock hardware, or custom hardware with lots of TPUs? Our customers want all their video processing to happen on site, and don't want their video or other data to touch the cloud, so they're not happy about renting cloud TPUs or GPUs. Also it would be nice to have smart cameras with built-in TPUs.
- Yes, but what's at the finish line? The bottom?
by kittikitti
0 subcomment
- You can't really buy a TPU, you have to buy the entire data center that includes the TPU plus the services and support. In Google Colab, I often don't prefer the TPU either because the documentation for the AI isn't made for it. While this could all change in the long term, I also don't see these changes in Google's long term strategy. There's also the problem with Google's graveyard which isn't mentioned in the long term of the original article. Combined with these factors, I'm still skeptical about Google's lead on AI.
by danishSuri1994
0 subcomment
- [dead]
by Jeff-Collins
0 subcomment
- [dead]
- Right because people would love to get locked into another even more expensive platform.
- https://killedbygoogle.com
- This is the “Microsoft will dominate the Internet” stage.
The truth is the LLM boom has opened the first major crack in Google as the front page of the web (the biggest since Facebook), in the same way the web in the long run made Windows so irrelevant Microsoft seemingly don’t care about it at all.
- They can only privatize the AI race.
If Google wins, we all lose.
- How high are the chances that as soon as China produces their own competitive TPU/GPU, they'll invade Taiwan in order to starve the West in regards to processing power, while at the same time getting an exclusive grip on the Taiwanese Fabs?
- In my 20+ years of following NVIDIA, I have learned to never bet against them long-term. I actually do not know exactly why they continually win, but they do. The main issue they have a 3-4 year gap between wanting a new design pivot and realizing it (silicon has a long "pipeline"), it can seem that they may be missing a new trend or swerve in the demands of the market, it is often simply because there is this delay.
Hacker News