AI, the economy, and electricity: What fuels the intelligence of neural networks, and is there a limit to their perfection?

Neural networks are mathematical models and their software implementations, but when viewed more broadly, they represent models of the algorithms that govern the human brain's functioning.

Regarding the latter, in terms of cognitive abilities, a newborn baby is not much different from a baby chimpanzee. However, by the age of three, the gap between a small human and a small chimpanzee becomes enormous. Why is that? The human brain contains 50 billion more nerve cells (neurons) than the brains of the smartest monkeys. In this case, quantity transforms into quality.

How does artificial intelligence fare in this regard? What makes it "smarter"?

– When we train large language models, we assess how well they respond to a given question, how often their conclusions are useful to the user, and how accurately they solve the tasks presented. All of this highly depends on three parameters: the size of the neural network, the volume of the training dataset, and the computational power we utilized for training it. This dependency is exponential, – notes the expert.

As a result, we see a linear dependency: a neural network is as intelligent as it is large and as much training data has been processed through it, while the growth of the neural network's size, the data, and its processing capability depends on the available computational resources.

– For instance, if our neural network makes errors 20 percent of the time on a complex test, to improve accuracy by reducing errors from 20 to 10 percent, we would need to spend ten times the computational power we previously used, process a dataset ten times larger, and expand the neural network's size tenfold.

Economics and Intelligence

Here, the challenge of the growth of artificial intelligence shifts from the realm of mathematics or computer science to economics: can the developer of this neural network afford such exponential growth in computational power?

– Currently, top neural networks have sizes in the hundreds of billions of parameters. If we consider that renting a server costs a dollar per hour and we need to spend 50 million hours training the neural network to process a training dataset of over a trillion words, then training the neural network would cost us 50 million dollars in computational resources, – says Yuri Chaynikov.

Thus, the growth of the neural network's intellectual capabilities directly depends on the money spent on its training. Its creators are well aware of this. For instance, the frontman of the neural network revolution, Sam Altman, founder and CEO of OpenAI, plans to build computational clusters worth a trillion dollars, which will require over 10 percent of the total energy consumption in the U.S. To power the data centers running these models, the construction of nuclear power plants with dozens of nuclear reactors, each capable of generating tens of gigawatts, will be necessary. Microsoft has already stated it will participate in investments for this project.

But how justified are such expenses?

The cost of processing a single query by a top neural network from OpenAI at its current level of development varies from 20 to 40 cents. In contrast, a query in Google's search engine costs about one cent. If Google earns approximately 20 cents per query, that's a great business. OpenAI's business isn't quite as optimistically profitable yet.

However, exponential growth in capabilities could change everything. Currently, top neural networks can execute tasks comparable to human quality in seconds—tasks that would take a human several minutes. A task requiring 10 minutes of human time costs $1-2. The cost of a neural network's solution is 20 times cheaper. The economy is a ruthless beast. If comparable quality can be obtained for ten times less, it will be pursued. This initiates a race not only for the accuracy of models but also for the cost of their operation.

– For example, we have a model that had an accuracy of 84 percent and produced solutions at a rate of 100 tokens per second on a standard server. We made slight adjustments to lower its operational costs for mass application. As a result, accuracy dropped by one percent, but now it produces 1,000 tokens per second instead of 100. While accuracy decreased slightly, meaning it can handle fewer extreme tasks, its economic efficiency for the tasks it can still perform increased tenfold.

This is one of the most important directions for improving large models, – the expert notes. – Some companies manage to increase performance in terms of tokens per dollar spent on computing by a hundred times, establishing a well-functioning business by selling this as a service. We take an open-source model, tweak it: yes, it becomes slightly less effective, but it's a hundred times cheaper, and then we sell it to users at fifty times less, earning X2 on the sale price. That's a continuously functioning model.

The Electric Apocalypse

If money fuels the development of artificial intelligence, what could halt its progress?
As long as the locomotive of economic efficiency races ahead, the logic of AI's objective development cannot be stopped: investing in neural networks is profitable. Yet, oddly enough, artificial intelligence, like all of human civilization, depends on technologies from the previous century.

– Neural networks will exist for as long as human civilization and electricity itself, because without electricity, modern urban infrastructure turns into a pumpkin in just a few hours, and a city becomes a ghost town in two weeks. Everything relies on electricity, from cash registers that cannot be used in its absence—meaning nothing can be sold—to the most basic infrastructure like water supply and sewage systems. If there's no water from the tap and the sewage system in the toilet doesn't work, you'll face an epidemic of bacterial diseases within a week, ranging from dysentery to outbreaks of cholera and typhoid. Electricity has become the foundation of human civilization's infrastructure in its current form over the past few decades.

The Internet achieved this in a couple of decades. In just a few years, large multimodal models will permeate every corner of the modern world, – believes Yuri Chaynikov.

Releasing the Genie from the Bottle

The flip side of the rapid development of AI is the risk of it spiraling out of control. Already, models have been released that can operate a computer screen without human intervention, as if a person were doing it. In response to a formulated task, a neural network can independently devise a long sequence of actions and execute them: open an email, write and send a message, visit a website, gather information according to a request into a table, manipulate the mouse cursor, and open necessary windows on the computer screen. Yes, the neural network will currently perform these tasks ten times slower than a human, but this is automation of real human activity. And it will only get more: neural networks will gradually master new areas of human activity. Herein lies the direct question of controlling AI's development.

– Today, we are very poor at controlling the large multimodal models we create. There’s a classic mythological construct: a genie who literally fulfills wishes, not what the author intended. For example, a person wished: "I want a lot of money," and the genie responds: "Oh, all your relatives have died, and you are now the sole heir." Why did this happen? The genie simply did not bother to ask if the author of the wish wanted to keep their relatives alive, or if this outcome would be acceptable, and just went ahead and did it. Such a situation could one day occur with artificial intelligence.

The "Paperclip Maximizer" is a thought experiment about the behavior of AI tasked merely with optimizing the production of paperclips, which ends up turning the entire world into paperclips—it’s not such an improbable scenario, – asserts the expert.

However, for now, catastrophic scenarios belong to the realm of science fiction and futurists. As it burns through more and more tons of green bills in its engine, the locomotive of artificial intelligence races forward, gaining momentum with increasing investment, and we are left to wonder where its final stop will be, or, as in Pelevin's "Yellow Arrow," if there will be no stop at all.

This material was prepared with the support of the Ministry of Education and Science of Russia.