html
Zuckerberg highlighted that Meta's Llama 4 AI models are being trained on an unprecedented GPU cluster.
Meta
Mark Zuckerberg claims that Meta's Llama 4 AI models are utilizing the largest GPU cluster in the industry.
During a recent earnings call, he mentioned that this cluster exceeds "100,000 H100s."
Elon Musk has stated that his company xAI is employing 100,000 Nvidia H100 GPUs for training its chatbot Grok.
While Elon Musk has been vocal about his AI startup's extensive collection of Nvidia chips, Mark Zuckerberg is now showcasing Meta's capabilities.
A significant amount of computational power is dedicated to developing Meta’s upcoming Llama 4 AI models—more than what competitors currently possess, according to Zuckerberg.
In a Q3 earnings call held on Wednesday, the CEO of Meta revealed that Llama 4 is "well into its development," being trained on a GPU cluster larger than any other in the market.
"We’re training the Llama 4 models on a setup exceeding 100,000 H100s or anything else I’ve seen reported from our competitors," he stated confidently.
This figure of 100,000 may reference Musk’s xAI initiative which recently unveiled its "Colossus" supercomputer. The Tesla founder has described it as the world's largest supercomputer and confirmed it utilizes an impressive array of Nvidia’s H100 graphics processing units (GPUs) for training Grok.
Nvidia’s H100 chip—also referred to as Hopper—is highly coveted among technology leaders and AI startups due to its exceptional computing power for large language model training. Each chip carries an estimated price tag between $30,000 and $40,000.
The accumulation of these powerful chips plays a crucial role in attracting top-tier talent within the AI sector. Aravind Srinivas, CEO of Perplexity.ai, shared during a podcast how this topic arose when he attempted to recruit someone from Meta:
"I tried to hire a senior researcher from Meta; their response was: 'Come back when you have at least 10,000 H100 GPUs,'" Srinivas recounted in March.
Meta previously launched its Llama 3 models earlier this year in April and July. During Wednesday's earnings call, Zuckerberg noted that Llama 4 will introduce "new modalities and capabilities with enhanced reasoning," promising significantly faster performance. He anticipates smaller model versions could be ready for release by early 2025.
When questioned about Meta's substantial investments in artificial intelligence infrastructure development, Zuckerberg acknowledged that they are expanding their capabilities more rapidly than initially projected. He expressed satisfaction with his team's execution despite acknowledging higher costs might not align with investor expectations:
"We're building out our AI infrastructure quicker than anticipated; I'm pleased with how well our team is performing even if it leads to increased expenses," he remarked during the call.
The company expects capital expenditures related to scaling up their AI infrastructure will continue rising into next year. While specific details regarding the size of their H100 chip cluster were not disclosed by Zuckerberg directly; Musk tweeted earlier this week indicating plans for xAI to double its current capacity soon—aiming for around 200 thousand combined H100 and H200 chips within months ahead.
