The race is underway to achieve Artificial General Intelligence (AGI), where the system understands, learns, and creatively performs most tasks equal to or superior to humans.
Some experts predict we will reach AGI within a decade — or sooner. But hold off polishing the medals. Reaching the AGI finish line requires overcoming significant hurdles along the way.
Let’s tour the track to see what we’re up against.
Moore Like A Rule of Thumb
Compute demands of AI are growing at rates that are unsustainable without a paradigm shift in underlying technologies.
As Nick Harris, CEO of Lightmatter, states:
“The amount of [computing power] that AI needs doubles every three months…It’s going to break companies and economies.”
Harris’ assertions are echoed by Epoch AI, which finds training compute for new AI models is growing up to 5x per year.
Can industry keep pace with this demand?
In 1965, Gordon Moore, the co-founder of Intel, postulated that the number of transistors in semiconductor chips would double every two years.
His observation became known as Moore’s Law, which reliably predicted the growth in computing power for more than 50 years.
It turns out the ‘Law’ was never a scientific principle but rather a rule of thumb that is now bumping up against actual laws of physics as miniaturization approaches the atomic level.
Dr. Moore foretold his namesake’s physical and economic limits in his 2007 statement:
“In a decade and a half, I think we’ll hit something fairly fundamental.”
James Powell gave scientific rigor to the argument with his 2008 calculation that Moore’s Law will be obsolete by 2036 due to the Heisenberg uncertainty principle.
Will the demise of Moore’s Law be a roadblock to AGI?
Intel and others are working feverishly to extend Moore’s Law using 3D chips, photonics, and materials that are just 3 atoms thick. These efforts may achieve a trillion transistors in a package by 2030.
Even if we achieve these goals, processor density is only one of the compute hurdles along the path to AGI.
Parallelism Paralysis
Research by Nvidia and Microsoft shows that if GPT-3, with a modest 175 billion parameters, had been trained on a single Nvidia V100 GPU, it would have taken a ridiculous 288 years to complete.
Training larger AI models like GPT-4 and GPT-5 on a single processor would take thousands of years!
The solution to this single processor bottleneck was to run massive parallel processor farms with 25,000 GPUs or more, splitting the problem into chunks and reassembling the results into the final product.
For GPT-4, training with parallel processing took just 95 days compared to 6,500 years if it had been trained on a single processor.
Still, parallel processing has challenges.
The problem arises in the connections among parallel components. As Professor Mark Parsons, director of supercomputing at the University of Edinburgh, describes:
“Even if the GPUs can become faster, the bottleneck will still exist as the interconnectors between GPUs and between systems aren’t fast enough.”
NVIDIA is progressing on this bandwidth problem with its DGX™ GH200 Supercomputer by combining GPUs and massive memory in one package that increases bandwidth by 7x.
Combined with more efficient parallel processing algorithms, this hardware could pave the way for language models with up to 80 trillion parameters, which may be enough to achieve basic levels of AGI.
How Many Light Bulbs Does It Take…
Growing demands for powering and cooling millions of AI processors are raising environmental impact concerns.
As one example, training GPT-3 consumed more than one million gallons of water, and processing its daily user requests requires 1 gigawatt hour of electricity — enough to power 33,000 US households.
AI’s water consumption is projected to reach 6.6 billion cubic meters by 2027 — or half the annual water consumption of the United Kingdom. In the same timeframe, AI’s electrical demand may reach 134 terawatt-hours per year — or 0.5 percent of the entire globe’s energy demands.
Meeting these numbers will not be cheap. US utilities must invest $50 billion in new generation capacity for data centers alone.
In addition, analysts expect data center power consumption in the US will drive around 3.3 billion cubic feet per day of new natural gas demand by 2030, which will require new pipeline capacity.
These projections raise concerns in a world experiencing regional shortages of fresh water and desperately trying to transition from fossil fuels to renewables.
Hopefully, AI-derived solutions will help offset AI’s resource consumption through improved efficiencies in power grids, transportation, manufacturing, and other industries.
Perhaps this will spur us to redouble our efforts to slow climate change.
Junk Food On The Training Table
Training AI requires vast troves of high-quality data in the form of text, images, and video; the more data, the more capable the AI becomes.
This “scaling law” has suddenly hit multiple snags that may become disqualifiers in the AGI race.
Epoch AI Research Institute projects with 80% confidence that the human-generated data stock will be fully utilized between 2026 and 2032.
This may be an optimistic prediction as the study did not anticipate several speed bumps affecting access and quality.
Big Tech has been grazing freely on available internet data to feed their insatiable AI models for years, often ignoring ownership rights or modifying the small print in their service agreements to claim rights.
Content creators have begun fighting the questionable scraping of their property with lawsuits, web bot restrictions, paywalls, and other restrictive measures.
A recent analysis by the MIT-led Data Provenance Initiative examined the Common Crawl repository used to train GPT and found that 45%+ of actively maintained, critical sources are now restricted.
Big Tech has reacted by cutting deals with publications to gain access to their data.
OpenAI spends millions on partnerships to access content from The Atlantic, Vox Media, The Associated Press, the Financial Times, Time, and News Corp.
Still, these deals are only band-aids. A comprehensive solution will be elusive. As Sy Damle, representing venture capital firm Andreessen Horowitz, put it:
“The data needed is so massive that even collective licensing really can’t work.”
The shortage of quality data is exasperated by a flood of AI-generated content polluting the training data pool.
A recent study by AI detector firm Copyleaks analyzed 100 million web pages 17 months after the release of ChatGPT 3.
Their findings revealed that 1.57% contained AI-generated content, most coming from legitimate sources seeking productivity gains from AI.
However, not all AI-generated content is benign.
A report by Google DeepMind concludes we may continue to see an increase in AI-generated content as part of misinformation and manipulation campaigns.
The fear is that future AI models will be trained on this tainted data feed.
The flow of AI-generated data will only get worse. As Copyleaks co-founder and CEO Alon Yamin explains:
“The demand for scalable and cost-effective content creation…has propelled the adoption of AI-generated content, driving its exponential growth on the web.”
It seems AI’s short-term success may be undermining its future.
Processed Food By Another Name
The cherry on top of all this havoc is Big Tech’s plan to feed synthetic data generated by AI into future models. How this will work is a puzzle.
Raw human-created data contains a multitude of variations in how original authors saw their world. This variation, which includes bias, gives us a diversity of views on topics ranging from philosophy to what a typical dog looks like.
Recent research on feeding synthetic data from one generation of AI to another revealed a phenomenon called “model collapse” — a degenerative process whereby, over generations, models forget the underlying variations in the data.
Model collapse appears fundamental to algorithms gravitating toward the most common answer. If you ask AI to create an image of a dog, it won’t give you a rare breed it only saw twice in its training data; you’ll probably get a golden retriever.
The problem of feeding synthetic data from one generation to another is that the final model forgets it ever saw rare breeds in its dataset.
Do we really want an AGI whose only flavor is vanilla?
Learning The Old Fashioned Way
Fei-Fei Li, a.k.a. the “Godmother of AI,” sees concerns about AI running out of data as a “narrow viewpoint” pointing out that:
“The health care industry is not running out of data, nor are industries like education.”
She has a point — up to a point.
Li offers a glimpse at the solution, saying:
“We need…more entrepreneurial exchange of information…imagining how to communicate to and with tech, biotech, teachers, doctors…there are people out there thinking in the most imaginative ways.”
The fundamental problem is that there will always be a finite amount of human-generated data.
Overcoming this constraint requires the system to gain AGI Agency, whereby robotics and humans act as real-world agents, allowing AGI to acquire new knowledge the old-fashioned way through experimentation and observation.
Thomas Edison tried and failed 2,774 times before finding his design for the electric light bulb. His effort echoes the trial and error method used in many breakthroughs in science, engineering, technology, medicine, healthcare, and business.
For AGI to become super-intelligent, it must team with humans to test its hypotheses and learn from its mistakes.
First blushes of the human/AI teaming approach are underway.
A good example is AlphaFold, whose open database of 200 million AI-predicted protein structures is being used worldwide by scientists researching industrial and medical solutions, such as plastics recycling and cancer cures.
Other examples show how AI Agency can operate in the real world with minimal human involvement.
In marketing, human-driven A/B testing of advertising copy is being revolutionized by AI, which designs ads, measures response, and iterates toward optimal effect. Which AGI will come up with the best jingle?
The same applies to AI-driven stock trading, in which AI devises and runs investment portfolios based on historical and real-time market and sentiment data feeds. Are tips from other AI Traders considered insider information?
The field is wide open for similar opportunities where response data is readily available, and AI has the agency to test and learn.
Extending the AGI Agency principle requires caution but may be the only long-term way to accelerate new knowledge generation.
And, They’re Off…
Winning the race to Artificial General Intelligence (AGI) has significant but surmountable hurdles, including escalating computing demands, the limits of Moore’s Law, parallel processing complexity, and environmental impacts.
Data scarcity and the risks of synthetic data further hinder the path. Inevitably, leveraging human-AI collaboration and AGI Agency is the way forward.
There will be drama.
The lead will change many times. Claims of victory will be made and challenged. Records will be set and quickly broken.
In the end, we hope to crown winners that balance technological advancements with ethical and sustainable practices to ensure AGI benefits humanity.
