The Age of giant artificial intelligence models is already over, says the CEO of Openai
Artificial Intelligence: What will the next generation learn from openAI? Spokesman Sam Altman says GPT-4 isn’t going anywhere
There is a surge of new interest and investment in artificial intelligence, because of the stunning capabilities of the chatbot from startup Openai. But late last week, OpenAI’s CEO warned that the research strategy that birthed the bot is played out. It’s unclear exactly where future advances will come from.
According to Altman, GPT-4 could be the last major advancement from OpenAI’s strategy of feeding the models more data. He did not say what kind of research strategies or techniques might take its place. In a paper on GPT-4, OpenAI says its estimates suggest diminishing returns on scale up. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.
Sam Altman, the company’s CEO, said that models bigger wouldn’t make further progress. At an event at MIT late last week, he said that he thought the era of giant, giant models was coming to an end. “We’ll make them better in other ways.”
Nick Frosst, a cofounder at Cohere who previously worked on AI at Google, says Altman’s feeling that going bigger will not work indefinitely rings true. He, too, believes that progress on transformers, the type of machine learning model at the heart of GPT-4 and its rivals, lies beyond scaling. He says that adding parameters to the model is not one of the ways of making transformers way, way better and more useful. Frosst says that new AI model designs, or architectures, and further tuning based on human feedback are promising directions that many researchers are already exploring.
Bubeck went to his computer and asked GPT4 to use TikZ, an obscure programming language which can be used to generate scientific diagrams. The GPT-4 version was only used for text and not images. When fed a TikZ rendering software, the code presented to him by the model conjured up a crude but bright image cobbled together from multiple objects. To Bubeck, a feat like this requires an abstract grasp of the elements of a creature. “Something new is happening here,” he says. “Maybe for the first time we have something that we could call intelligence.”
How intelligent AI is becoming—and how much to trust the increasingly common feeling that a piece of software is intelligent—has become a pressing, almost panic-inducing, question.
