Google unveils chips for AI training and inference in latest shot at Nvidia

After years of making chips capable of both training AI models and performing inference, Google is now splitting those functions into separate processors, marking its latest move to challenge Nvidia in the AI hardware space.

Google said Wednesday that it’s making the change for the eighth generation of its tensor processing unit, or TPU. Both chips will become available later this year.

“With the rise of AI agents, we determined the community would benefit from chips individually specialized to the needs of training and serving,” Amin Vahdat, a Google senior vice president and chief technologist for AI and infrastructure, said in a blog post.

In March, Nvidia highlighted upcoming chips designed to help models quickly respond to user questions, using technology gained from its $20 billion acquisition of chip startup Groq. While Google is a major Nvidia customer, it also offers its own TPUs as an alternative for companies using its cloud services.

Many of the world’s leading tech companies are diving into custom semiconductor development for AI to boost efficiency and create chips tailored for specific needs. Apple has been adding neural engine components to its in-house iPhone chips for years. In January, Microsoft revealed its second-generation AI chip, and just last week, Meta announced a partnership with Broadcom to develop several versions of AI processors.

Google jumped on the trend early, designing its own processors for running AI models in 2015 and starting to rent them out to cloud clients by 2018. That same year, Amazon Web Services introduced the Inferentia chip for handling AI requests, followed by the Trainium processor in 2020 for training AI models.

In September, DA Davidson analysts estimated that the TPU business, combined with the Google DeepMind AI group, could be valued at around $900 billion.

None of the big tech players are unseating Nvidia, and Google isn’t even stacking its new chips against the AI chip leader’s. However, Google mentioned that its training chip delivers 2.8 times the performance of the seventh-generation Ironwood TPU, unveiled in November, at the same price, while the inference processor offers an 80% performance boost.

Nvidia announced that its upcoming Groq 3 LPU hardware will use large amounts of static random-access memory (SRAM), a technology also employed by Cerebras, the AI chipmaker that filed to go public earlier this month. Google’s new inference chip, the TPU 8i, likewise depends on SRAM, with each chip packing 384 megabytes—three times the amount found in Ironwood.

Sundar Pichai, CEO of Google’s parent company Alphabet, wrote in a blog post that the architecture is built to provide the massive throughput and low latency required to run millions of agents at the same time, all in a cost-effective way.

Google’s AI chips are gaining traction fast. Citadel Securities has developed quantitative research software powered by Google’s TPUs, and all 17 U.S. Energy Department national labs are running AI co-scientist software built on them. Meanwhile, Anthropic has pledged to use multiple gigawatts of Google TPUs.