As Alphabet Inc (GOOGL.O) overlooks a chatbot error that helped wipe off $100 billion from its market value, another challenge is cropping up from its efforts to embed generative artificial intelligence into its popular Google Search: the cost.
Executives across the technology sector are discussing how to operate AI like ChatGPT while taking into account the high expense. The wildly acclaimed chatbot from OpenAI, which can write prose and answer search queries, has “eye-watering” computing costs of a few or more cents per conversation, the startup’s Chief Executive Sam Altman has said on Twitter.
In an interview, Alphabet’s Chairman John Hennessy told Reuters that having a dialogue with AI called a large language model probably cost 10 times more than a standard keyword search, though fine-tuning will help cut down the expense quickly.
Even with revenue from potential chat-based search ads, the technology could chip into the net income of Mountain View, Calif.-based Alphabet with several billion dollars of additional costs, analysts said. Its net income was nearly $60 billion last year.
Morgan Stanley determined that Google’s 3.3 trillion search queries in 2022 cost almost a fifth of a cent each, a number that would grow based on how much text AI must generate. Google, for example, could face a $6-billion increase in expenses by next year if ChatGPT-like AI were to handle half the queries it gets with 50-word answers, analysts predicted. Google is not expected to need a chatbot to handle navigational searches for sites like Wikipedia.
Others worked out a similar bill in different ways. For example, SemiAnalysis, a research and consulting company focused on chip technology, said integrating ChatGPT-style AI to search could cost Alphabet $3 billion, an amount limited by Google’s in-house chips called Tensor Processing Units, or TPUs, together with other optimizations.
What makes this form of AI more costly than conventional search is the computing power required. Such AI relies on billions of dollars of chips, a cost that has to be distributed over their useful life of several years, analysts said. Electricity also adds costs and pressure to firms with carbon-footprint goals.
The process of handling AI-powered search queries is called “inference,” in which a “neural network” loosely modeled on the human brain’s biology deduces the answer to a query from prior training.
In a traditional search, by comparison, Google’s web crawlers have scoured the internet to put together an index of information. When a user types a query, Google provides the most relevant answers stored in the index.
Alphabet’s Hennessy told Reuters, “It’s inference costs you have to drive down,” calling that “a couple-year problem at worst.”
Alphabet is under pressure to take on the challenge despite the expense. Earlier in February, its rival Microsoft Corp (MSFT.O) hosted a high-profile event at its Redmond, Washington headquarters to show off plans to imbue AI chat technology into its Bing search engine, with senior executives targeting Google’s search market share of 91%, by Similarweb’s estimate.
A day later, Alphabet discussed plans to improve its search engine, but a promotional video for its AI chatbot Bard showed the system answering a query incorrectly, prompting a stock plunge that erased $100 billion off its market value.
Microsoft later attracted scrutiny of its own when its AI reportedly threatened or professed love to test users, causing the company to limit long chat sessions it said “provoked” unintended answers.
Microsoft’s Chief Financial Officer Amy Hood has told analysts that the benefit from gaining users and advertising revenue exceeds expenses as the new Bing launches to millions of consumers. “That’s incremental gross margin dollars for us, even at the cost to serve that we’re discussing,” she said.
Another Google rival, CEO of search engine You.com Richard Socher, said embedding an artificial intelligence chat experience along with applications for charts, videos, and other generative tech increased expenses between 30% and 50%. “Technology gets cheaper at scale and over time,” he said.
A source close to Google warned it’s early to identify exactly how much chatbots might cost because efficiency and usage vary broadly depending on the technology involved, and AI already powers products like search.
Still, paying the bill is one of two primary reasons why search and social media titans with billions of users have not introduced an AI chatbot overnight, said Paul Daugherty, Accenture’s chief technology officer.
“One is accuracy, and the second is you have to scale this in the right way,” he said.
Making The Math Work For Artificial Intelligence
For years, researchers at Alphabet and elsewhere have delved into how to train and run big language models more inexpensively.
Larger models need more chips for inference and therefore cost more. AI that fascinates consumers for its human-like authority has increased in size, reaching 175 billion so-called parameters, or different values that the algorithm takes into consideration, for the model OpenAI updated into ChatGPT. Cost also differs by the length of a user’s query, as measured in “tokens” or pieces of words.
One top technology executive told Reuters that such AI remained too expensive to put in millions of consumers’ hands.
“These models are very expensive, and so the next level of invention is going to be reducing the cost of both training these models and inference so that we can use it in every application,” the executive said on condition of anonymity.
For now, computer scientists inside OpenAI have discovered how to optimize inference costs through complex code that makes chips operate more efficiently, a person conversant with the effort said. An OpenAI spokesperson failed to immediately comment.
A longer-term challenge is how to reduce the number of parameters in an AI model 10 or even 100 times, without losing accuracy.
Buy Crypto Now“How you cull (parameters away) most effectively, that’s still an open question,” said Naveen Rao, who previously ran Intel Corp’s AI chip efforts and currently works to shrink AI computing costs through his startup MosaicML.
In the meantime, some have considered billing for access, like OpenAI’s $20 per month subscription for better ChatGPT service. Technology experts also said a workaround is using smaller artificial intelligence models for simpler tasks, which Alphabet is considering.
The company said in February a “smaller model” version of its massive LaMDA AI technology will operate its chatbot Bard, requiring “significantly less computing power, enabling us to scale to more users.”
Asked about chatbots like ChatGPT and Bard, Hennessy said at a conference known as TechSurge last week that more focused models, instead of one system doing everything, would help “tame the cost.”
