(Bloomberg Businessweek) -- The two years since OpenAI supercharged the generative AI era with the introduction of ChatGPT have passed in a blur of technological one-upmanship. OpenAI and its primary competitors, Anthropic, Google and Meta, have released a flurry of cutting-edge artificial intelligence models, each more skillful than the last. It’s now Silicon Valley gospel that more computing power, more data and larger models will lead to such fundamental improvements in AI that the technology will transform entire industries within the next few years.
And yet, threats to the pace of development began emerging even before ChatGPT’s second birthday. In 2024, OpenAI and two other leading AI companies hit stumbling blocks. At OpenAI and Google, some software failed to live up to internal expectations, while the timetable of a long-awaited model from Anthropic, a competitor built by former OpenAI employees, slipped after it had already been announced. If progress in generative AI slows in some durable way, it will bring into question whether the technology can ever achieve the more expansive promises the industry’s top innovators have made for it. Identifying ways to propel the AI boom into its next stage will be the field’s primary challenge in 2025.
The companies are facing several hurdles. It’s become harder to find new sources of high-quality, human-made training data to build more advanced AI systems. In addition, even modest improvements in AI performance may not be enough to justify the tremendous costs associated with creating and operating new models. Dario Amodei, Anthropic’s chief executive officer, has said it costs about $100 million to train a bleeding-edge model, and he expects that amount to hit $100 billion in the coming years. OpenAI Chief Financial Officer Sarah Friar says that it would be fair to say the company’s next cutting-edge model will cost billions to develop, and that there’s still a need for “bigger and bigger models that are more and more expensive.”
These issues raise doubts about the billions of dollars already invested in AI and about the goal these companies are aggressively pursuing—so-called artificial general intelligence, or AGI, which could match or best humans on a wide range of tasks. The chief executives of OpenAI and Anthropic have previously said that AGI may be only several years away, and both have dismissed any suggestion that they’re hitting a wall. Other industry leaders, though, strike a more humble tone. “I think the progress is going to get harder,” Google CEO Sundar Pichai said during an interview at the New York Times’ DealBook Summit in early December. Looking to 2025, he said, “the low-hanging fruit is gone, the hill is steeper.”
In an industry that prides itself on innovation, companies are looking for different ways to push AI models forward. Efforts underway include getting computers to mimic how humans mull over a problem to better solve it, building models that are really good at certain kinds of tasks and training artificial intelligence with data generated by AI itself.
OpenAI, in particular, has been an early proponent of AI that can perform humanlike reasoning to take on more complex queries and improve over time, particularly when it comes to questions related to math, science and coding. In September the company unveiled an early version of a model called o1 that does this by spending more time computing an answer before responding to a user’s question. OpenAI announced an improved version of that model in December, which CEO Sam Altman referred to on social media as “the smartest model in the world.”
The company is so confident about o1 that it’s begun offering a $200 monthly subscription that includes, among other features, access to a version of the model that can use even more computing power to answer questions.
Several other companies, including Google and software maker Databricks, are working on their own versions of this approach, which is often referred to as test-time or inference-time compute. Jonathan Frankle, Databricks’ chief AI scientist, says he expects to see the technique become much more widespread in the industry. In addition to providing better answers, he says, it could improve the economics of building AI models by repositioning some costs from prerelease development to times when the models are already in use and thus generating revenue.
Technological rumination doesn’t solve the issue of AI’s ever-growing hunger for data. Companies are increasingly turning to synthetic data, which can take many forms, including that of computer-generated text meant to mimic content created by real people. Nathan Lambert, a research scientist at the Allen Institute for AI, says that when developing a model called Tulu 3, he and his colleagues prompted an AI model to produce questions based on certain personas. They’d ask it, for instance, to pretend to be an astronaut and devise a math problem specific to that occupation, to which it would produce a question about how far the moon will be from the sun at a certain time of day. They’d feed the questions back into the model, then use both the questions and the answers to fine-tune their own system. For some reason, this technique improved Tulu 3’s math capabilities. “We don’t know why it works fully,” Lambert says, “and that’s the exciting side of synthetic data.”
Large language models—the kind of AI software that powers ChatGPT—are meant to replicate the words humans use to communicate, so simply training an AI system on the content it produces won’t lead to improvements, according to Frankle. Lambert says it’s important to filter AI-generated data to avoid repetition and verify that it’s accurate. Some researchers have also raised concerns that indiscriminately using such data for training could hamper a model’s performance (a consequence they referred to as “model collapse”).
Fei-Fei Li, co-director of Stanford University’s Institute for Human-Centered AI and a co-founder of the AI startup World Labs, expects AI companies to rely increasingly on synthetic data. She pointed out that those working on self-driving car technology, for instance, have long relied upon simulated driving data. “In the tech stack of AI, data is as important as algorithms,” Li says. “Synthetic data has a huge role to play.”
The problems of scaling up enormous general-purpose models are particularly relevant if the goal is to work toward AGI. But historically, AI has instead been designed to focus on a single task, and Databricks’ Frankle says there’s plenty of room to innovate there. Overall, he’s optimistic. He likens what’s happening in the AI industry now to what’s happened in the evolution of the chip industry, where chip developers would reach what looked like a hard limit, then come up with different innovations—multicore processors, co-processors, parallel processing—to keep improving the technology.
“Looking back at our semiconductor days,” he says, “you go from one innovation to the next and just keep trying to push forward.” —With Shirin Ghaffary
©2024 Bloomberg L.P.