Certain definitions of AGI are backing people into a pseudo-religious corner.
Artificial general intelligence (AGI) doesn’t need to involve the idea of agency. The term’s three words only indicate a general level of capability, but given its lack of grounding in any regulatory agency or academic community, no one can control what others think it means. The biggest problem with the AGI debate is different folks have different end goals and value systems around the symbol that AGI represents. These end goals have added new constraints on the Overton window of acceptable AGI definitions towards more advanced capabilities.
The origins of the discussion of AGI are actually quite benign, and recent. It replaced a previous definition, Strong AI, to encompass something more vague, as strong AI could be achieved in many ways. From Wikipedia:
The term "artificial general intelligence" was used as early as 1997, by Mark Gubrud in a discussion of the implications of fully automated military production and operations. A mathematical formalism of AGI was proposed by Marcus Hutter in 2000. Named AIXI, the proposed AGI agent maximises “the ability to satisfy goals in a wide range of environments”. This type of AGI, characterized by the ability to maximise a mathematical definition of intelligence rather than exhibit human-like behaviour, was also called universal artificial intelligence.
The framework AIXI is described on Lesswrong (the popular AI Alignment discussion board) as “not computable, and so does not serve as a design for a real-world AI.” A more modern definition of AGI could be what OpenAI defines AGI as:
highly autonomous systems that outperform humans at most economically valuable work.
This sort of definition is surely what is convenient for them and not the only acceptable definition of AGI. I’m actually okay with calling a system like ChatGPT-4 AGI, but that loops me in with the hype machine that was the Sparks of AGI paper from Microsoft:
Given the breadth and depth of GPT-4’s capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system.
The claim above is actually quite reasonable if accompanied with a sufficiently mellow definition of AGI as an AI system that can match human capabilities on a broad suite of tasks. Regardless of what the formal definition is, it’s clear that GPT-4 fits many colloquial definitions of AGI. My definition has always been quite literal: an AI system that is generally useful.
The media backlash following the Sparks of AGI paper largely stems from the methods in the paper and definitions of AGI relative to OpenAI. Given OpenAI’s dominance of the mindshare for what AI is, and the specific importance of AGI to their company charter and their relationship with Microsoft, the AI community largely struck out against the authors as over-claiming. The paper itself read more as a list of examples and a user manual for GPT-4-Vision, built on non-public APIs, rather than an argument for what is or is not AGI. If the Sparks of AGI paper had made this case for what AGI is rather than doing marketing for OpenAI, none of the blowback would have occurred.
Recall that once OpenAI achieves AGI, Microsoft’s market position will weaken substantially. From VentureBeat:
once the [OpenAI] board decides AGI, or artificial general intelligence, has been reached, such a system will be “excluded from IP licenses and other commercial terms with Microsoft, which only apply to pre-AGI technology.”
This makes that entire debacle look even more silly. Microsoft’s researchers were making an extremely aggrandizing case of what is AGI that weakened their own marketplace. Satya, the CEO of Microsoft, was probably not thrilled.
My confusion as someone trained in reinforcement learning is over how this subfield of AI manages to still capture the cultural zeitgeist even when its technical solutions are largely out of style. The RL from human feedback (RLHF) revolution that happened with ChatGPT doesn’t change the truth that RL research passed a zenith of interest in the ML community in the timeframe of 2017 to 2020. It grounds me in remembering that every time someone is talking about AGI they’re talking about vibes rather than reality.
Regardless, the cultural narrative of reinforcement learning still is heavily impacted on the psyche of the machine learning community. The narratives of RL are more effective than the methods. Reinforcement learning researchers are obsessed with agency and feedback. The two terms together create an intriguing form of intelligence, but both of them are really needed. Feedback alone is just a while loop. Agency alone has no prospect of self-improvement or learning.
David Silver gave a tutorial on Deep RL in 2016 at ICML, where he had a slide that essentially claims that strong AI is at its essence a combination of RL and deep learning. My claim is that this attitude, even if further from the technical reality, is still the cultural reality.
The other form of RL’s AGI argument is the Reward is Enough paper. This mostly says, that as we approach the conditions discussed above on the limits of AGI, we can solve any problem with an RL-like approach. Seems like AGI is guaranteed then — it’s another mathematical formulation of AGI that assumes human society is static and we just have to drop an AI system and all else is static.
