On the Limits of Learned Intelligence
Will AI become smarter than humans, and if so, when?
This article is quite short and makes the following points:
The limit of intelligence for something which can modify itself is the information complexity of its environment
Humans are self-modifying intelligent learners and have direct sensory access to ‘reality’
AI models may become self-modifying, but do not currently have direct sensory access to ‘reality’
The set of sensory capabilities of an evolved organism use all available physical laws, while the sensory capabilities of a designed system only use the laws the designer understands
Therefore humans have a higher potential for intelligence than machines, until machines can evolve or we have a complete theory of physics.
Two Kinds of Knowledge
Suppose we can split knowledge into two kinds: observations, and relationships. In other words, if you can imagine a winged horse its only because you’ve seen wings and horses before. Wings and horses are both observations, putting them into the conjunction of ‘a horse that has wings’ is a relationship.
We acquire observations of the world through our senses, and construct models of the world to organize and explain these observations through relationships. This is ultimately a narrative constructive process: a way of telling a story that links the observations together in a meaningful way, where ‘meaning’ is really just ‘how satisfactory of an explanation did you think that story was of some underlying dynamic or causal mechanism.’ In that sense the most meaningful stories are theories from physics, which seem pretty good at explaining a huge number of observations with as minimal a story as possible.
The sum total of human knowledge is a set of observations and all the relationships that have been drawn between them, but, not all the relationships that could be drawn between them. If we adopt the principle of empiricism, then we should put more faith on the observations than the relationships - that is, our senses are less likely to deceive us than the stories we make up to explain them.
Self-Modifying Learners
The trick to doing science is being willing to modify the narratives you’ve accumulated to explain the observations you’ve had. This is usually quite difficult: we become emotionally attached to a specific way of seeing the world because the narratives we’ve accumulated have acted as justification for past investments, decisions, and our acceptance of those outcomes. Changing your belief system in the face of new information is very difficult and uncomfortable, which is why science doesn’t usually come naturally to human cultures.
The more you are able to update your narratives in the face of new information, the more accurate your narratives become in explaining the world. Ultimately you can explain more and more observations with less and less convoluted theories until you have a set of relatively simple theories that can interpret and explain almost all new observations.
This process of self-modification to get more elegant theories with greater explanatory depth is the process of becoming more data parsimonious with learning.
Evolved Senses versus Designed Senses
Biology is great - it uses all the available hacks in reality known as laws of physics to achieve its ends, because it explores the configuration space of possible designs without needing to understand how those designs work. Life, uh, finds a way.
Long before we had an accurate theory of quantum mechanics, cells used quantum processes to transform proteins, produce ATP, convert sunlight into sugar, and all sorts of other productive economic activity that drives shareholder value.
“All men love to know, and nowhere is that more apparent than the delight we take in our senses” wrote Aristotle. The senses we’ve evolved exploit all available physical laws to give us the power of observation, however, our abilities to construct narratives to explain these observations are limited by our understanding.
We’ve gotten really good at using observations to construct subtle and elegant theories of the world that has given us things like electronics, computers, cameras, horse drawn carriages and aqueducts. We can now use these technologies to create modern day carriages that see, hear, and to some degree think. But, the carriages we built are limited by the skill of the carpenters we’ve trained, ultimately they must understand the design they put forward while the evolved senses of a human require no such explanatory model, they simply evolve over time to use the physics available.
Therefore our observational powers essentially always exceed our explanatory powers to the extent we don’t have complete theories of the world that can explain all experiences. The risk we run is to throw away observations that don’t have good explanations - this would limit our ability to form new knowledge. It’s why hardline empiricism is ultimately the highest ideal in science. You have to explain your observations, even if it means throwing away your favorite theories.
The Problem with AI and Robots
The limitations of AI that is trained on human knowledge, or robots that learn from the environment, should now be quite obvious:
AI models like LLMs are trained on human knowledge, so they are ultimately limited to draw from the implicit set of observations in that knowledge base. Fortunately, they may be able to draw relationships far better than humans by incorporating more observations than a human could into ‘working memory’ - this can be especially advantageous in understanding a complex field like biology. Novel relationships drawn across fields may still be new knowledge, without requiring new observations.
Robots use senses that humans understand how to design, and so are limited by the extent of human knowledge. Still theres an advantage: we can make robots see in infra-red, ultraviolet, use laser interferometry, etc. In other words we can design senses that far exceed the human range, despite many human senses already operating at the limit of physical detection, e.g. your peripheral vision has single-photon detection abilities.
What does this mean in practice?
AI models can get smarter and smarter up to a limit - the limit of information implicit in their training environment. This still expands on human knowledge by going beyond whats explicit, or stated outright, by drawing new relationships across observations and linking them together. LLMs can therefore churn out vast quantities of new connectivist type knowledge that helps provide better narratives to explain things. A huge win, to be sure. And they can do it at-scale, on-demand, and far more cheaply than a person can. Massive productivity gain all around.
But, a self-modifying AI model doesn’t necessarily become infinitely smart, it just gets more data parsimonious over time. It can be smaller, more efficient, faster at learning the limits of information implicit in its environment. All good things for power consumption and API bills. But, it doesn’t mean you get an intelligence singularity.
Robots at least have direct senses to the environment and so can make new observations, but, their senses are things we can understand. We think we understand all our senses but we can’t even explain how brains work, therefore likely there are ways our bodies interact with the environment that we don’t fully understand but still manifest as features of our cognitive landscape that can provide information about the world around us.
Some Predictions
Therefore I’ll make a few predictions that might be totally wrong. I hope they are, because then I can update my entire world model to make it even more data parsimonious:
Text-based LLM competencies will naturally saturate at the level of comprehension of the world implicit in their training corpus, which might still manifest as new knowledge but only in the sense of drawing relationships across established observations. The biggest step-changes in LLM competency are already behind us. GPT5 will not be what GPT4 was to GPT3.
Even a self-improving LLM will only become more data parsimonious with its learning, and not lead to “infinitely improving intelligence,” because all it can do is exhaust the implicit knowledge in its training data.
Robots can acquire new observations about the world but won’t be able to employ the full sensory abilities of humans until we have a complete theory of physics and consciousness, or, we let robots evolve naturally such that we no longer need to understand how their senses work.
Of course, even with these limitations AI and robots can radically transform the world. An LLM can become smarter than almost everyone on the planet, robots can do essentially everything we need humans to do, etc. But, I no longer believe that we are facing an imminent AI singularity, nor will AI necessarily reveal all the deepest secrets of the cosmos.
Pushing back the final veil of night on the unknown to reveal the grandest truths might just require an experience of reality which depends on using those truths, in whatever way our bodies might use all available physics before understanding any physics. As always, the narrative comes after the observation, and as empiricists we can never let the narratives of today limit the observations of tomorrow.