AI chip company acquired by Nvidia for $20 billion

In February 2024, Groq CEO Jonathan Ross had a meeting in Oslo (Norway), where members of the Norwegian parliament and a series of technology executives appeared. Here, he demonstrated a groundbreaking product: an AI chatbot that can answer questions almost instantly, faster than human reading speed.

But the presentation did not go as expected. The chatbot ran slowly compared to expectations, making Ross worried, because this AI is operated by a data center in Europe using Groq chips. “I kept checking the numbers,” Ross told me Forbes. “People don’t understand why I’m so distracted.”

The “culprit” was then found: the number of new users flocking to the chatbot. One day before the show, a developer suddenly shared on

Theo CNBCNvidia is in the process of acquiring the assets of chip manufacturing startup Groq for $20 billion. According to the announcement on Groq’s blog, this is a non-exclusive agreement, in which Nvidia only takes the rights to use the company’s technology instead of taking over the entire thing, and the business can still continue to operate independently of the new CEO. And Ross, President Sunny Madras and other senior leaders joined Nvidia “to help develop and expand licensed technology.”

The power of the Groq chip

When videos appeared on social networks showing chatbots responding to users super quickly in early 2024, Groq was predicted to have the ability to challenge big names like Nvidia, AMD, Intel, while paving the way for new AI applications and models and accelerating the AI ​​race.

Theo Reuterscompared to Nvidia’s AI training chips that reuse GPU architecture, Groq goes its own way by building a completely new custom chip architecture called Language Processing Unit (LPU – Language Processing Unit), optimized for deterministic and single-token inference. Each LPU costs $20,000, equivalent to Nvidia’s A100 GPU.

One of the ways to evaluate the performance of AI hardware is the token generation rate per second (tps) when running the LLM model – an important index that reflects the ability to process natural language, the more tps, the faster and smoother the response from AI.

In May, Indian technology expert Prathisht Aiyappa published an analysis on Medium, comparing the Groq and Nvidia H100 chips and pointing out some differences. In particular, the Groq chip uses a token-based identification and processing mode, while the Nvidia chip uses a batch-oriented and probabilistic mode. Groq delivers around 300-500 tps with 1-2 ms latency, while Nvidia chips range between 60-100 tps and 8-10 ms latency. Groq’s own testing shows that, when running Llama 2 with 70 billion parameters, the token creation speed reaches 241 tps, more than double that of many other services on the market.

“When latency becomes a bottleneck affecting user experience, Groq’s advantages are maximized,” Aiyappa assessed.

Nvidia’s GPUs are considered the standard in the field of AI, both training and inference, thanks to their powerful parallel architecture platform and rich software ecosystems like Cuda and TensorRT. Meanwhile, Groq’s LPU chip can run most major language models today. However, this chip only achieves maximum performance in processing tasks related to text string inference. To train models, companies still need to use Nvidia GPUs or similar chips.

According to Aiyappa, Groq’s approach shows it’s not trying to “win at AI,” but instead is targeting a narrow but high-value market segment where Nvidia is underperforming: speed, determinism, and developer control. Besides, the product does not try to meet all needs, just focuses on building the fastest and most accurate inference tool, then providing it to groups that need it.

In fact, in the second quarter of 2024, Groq quietly signed contracts with many large and small companies and national organizations, mainly in the fields of real-time transcription, industrial robots, defense-level edge AI, health care… The company’s customers include Meta, Argonne National Laboratory (USA) used to research nuclear reactions, or oil and gas company Aramco Digital (Saudi Arabia).

“The ability to integrate the compiler with Groq hardware allows us to deliver products at speeds we cannot achieve using Nvidia or AWS solutions,” a CTO of a US-based AI company told CNBC.

“The Groq chip really targets the critical weakness,” Yann LeCun commented last year.

Founder’s vision

Ross is secretive and rarely appears in front of the media. According to his LinkedIn profile, he studied math and computer science at New York University’s Courant Institute of Mathematical Sciences. On Groq’s introduction page, Ross is one of Yann LeCun’s students – one of the people known as the Godfather of AI.

During the period 2006-2008, Ross studied for a doctorate at New York University but dropped out midway. Since 2009, he has held the position of head of R&D department of technology service provider Pacmid.

In 2011, Ross joined Google. As a software engineer, he created the distributed systems testing framework, one of the first key elements in building the Tensor Processing Unit (TPU) precursor platform for Google. TPUs are application-specific integrated circuits (ASICs) developed by Google that are used to accelerate machine learning workloads.

From 2013-2015, he focused on TPU and was in charge of designing and implementing core elements of the first generation chip. He then joined the Rapid Eval Team at Google X – Google’s secret lab, where he designs and incubates new projects for Alphabet.

During his time at Google, Ross wanted to design a chip model specifically for “reasoning”, with AI simulating human thinking and reasoning abilities by applying what it has learned to new situations. For example, a smartphone can identify a dog as a Corgi breed through photos that have never been seen before. This is different from the thinking of experts at that time, which was to train a giant model from the beginning.

In 2016, Ross and colleague Doug Wightman left Google to found Groq, building their own LPU chips. Wightman served as CEO for two years, then Ross took over. The same year, the company raised its first $10 million, with venture capital fund Social Capital leading the way.

In the following years, the company did not receive much attention, and Wightman left the company. A venture capitalist flatly refused when asked to raise capital because there was “no long-term potential”. Mitesh Agrawal, head of Lambda’s cloud computing department, also said he did not want to use Groq chips because “currently it is difficult to think beyond Nvidia”.

“Groq almost died many times,” Ross told Forbes last year. “Maybe we started a little early.”

However, the company still receives large amounts of money after capital raising rounds. By the Series C round in 2021, the company raised 300 million USD, was valued at more than a billion USD and became a unicorn,

At the end of 2022, OpenAI released ChatGPT, sparking a global AI fever, but luck has not yet come to Groq. The 2023 financial report shows that the company’s revenue reached 3.4 million USD, net loss was 88.3 million USD.

In early 2024, as the demand for computing power increases, the AI ​​semiconductor field begins to fragment. According to CNBCin the context of Nvidia dominating the computing chip market, other companies began to go in their own direction, such as TPU, FPGA (large integrated circuit using programmable logic element array structure), neural simulation chip… taking on specialized roles. The Groq chip is specifically designed for extremely low latency tasks. This helped them successfully raise capital in August 2024, raising 640 million USD, valued at 2.8 billion USD.

In September, Groq continued to successfully raise $750 million and was valued at $6.9 billion. However, the company’s journey is now turning in a new direction, when the finest technology and key personnel are transferred to Nvidia through an upcoming $20 billion deal.

According to observers, the deal will need to pass the final step, which is to be licensed by the regulatory agency. “The risk here is antitrust,” Bernstein analyst Stacy Rasgon wrote in a blog post.

As for Ross, he remained silent as usual. Last year, when honored on the Time 100 AI 2024 list, he said there is always a place for innovative companies. “The need is enormous,” he said. “When AI chips get cheaper, people will buy more.”