There is a weak argument that often crops up when we talk about the future of technology, especially at this moment in history when we talk about artificial intelligence: ‘Flying once seemed impossible; today we have aeroplanes. Even helicopters (adds the better-informed speaker)! Transplanting a heart seemed impossible; today we do it even in provincial hospitals. Humanity has always overcome its limits and will continue to do so.’ It is an appealing, optimistic and reassuring line of reasoning. Unfortunately, it is logically flawed.
The fact that humanity has achieved a great many things that seemed impossible before they were achieved does not prove that it can achieve them all. This is an unjustified induction: a universal conclusion is drawn from the observation of numerous particular successes.
David Hume had already identified the fundamental problem of induction: the fact that something has happened repeatedly does not, on its own, guarantee that it will continue to happen. Nor does it allow us to extend a regularity beyond what we have actually observed.
And for completeness: the history of science is not merely the history of the things we have managed to do. It is also the history of the things we have proved we cannot do. And some seemed so achievable that we kept trying.
Impossible things
For centuries, inventors and scientists tried to build a perpetual motion machine, capable of producing work without consuming energy. They failed. More importantly, though, we discovered why they could not succeed: such a machine would violate the laws of thermodynamics: it is a physical impossibility, according to the laws we know. We now know that even though humanity has managed to fly, this simply cannot be done. The same applies to the examples that follow.
Special relativity tells us that we cannot transmit information faster than light in a vacuum: the very structure of the theory imposes that limit.
For centuries, people sought a general formula, based on radicals, for solving fifth-degree equations, like those that exist for the second, third and fourth degrees. Abel and Galois proved that no general formula of that kind exists.
Alan Turing proved that there can be no universal algorithm capable of determining, for any program and any input, whether the program will terminate or continue indefinitely.
Kurt Gödel proved that every consistent formal system whose axioms can be enumerated by an algorithm, and which is powerful enough to express arithmetic, is necessarily incomplete: there are statements that can be neither proved nor disproved using the tools of that system.
In quantum mechanics, finally, there is no procedure capable of perfectly copying any unknown quantum state. Moreover, a particle cannot simultaneously have a position and a momentum both defined with unlimited precision.
These are not problems awaiting a better engineer, a more powerful computer, a few billion in funding, or time tending to infinity. They are limits we have discovered and defined through thought, mathematics and science. Accomplishing an extraordinarily difficult feat is impressive. But proving that a feat to which generations of scientists have devoted study and effort is impossible is perhaps equally so
To reason correctly, we must distinguish at least four categories.
Logical impossibility. For example, we cannot construct a round square while keeping the meanings of the words unchanged. The contradiction is internal to the very definition of the object.
Mathematical impossibility. We have seen that we cannot find an algorithm that solves the general halting problem.
Physical impossibility. We cannot build a machine that violates the second law of thermodynamics, assuming the validity of the physical laws currently known.
These physical impossibilities are, however, different in nature from mathematical ones: they depend on our theories about the world, which remain open to empirical revision.
Possibility yet to be established. This is the most interesting category, because it contains everything we do not know how to do today, but which we do not know how to prove impossible either.
It is this fourth category that is blithely treated as the waiting room of progress: ‘sooner or later, you’ll see…’
Something we do not know how to achieve today might be achieved tomorrow, in a hundred years, or never. But something else might also happen: we might discover a physical law, a logical contradiction or a mathematical proof that allows us to transfer it definitively, or at least within a well-corroborated physical theory, into the category of impossible things.
The progress of knowledge does not consist merely in making possible what we did not know how to do. It also consists in proving impossible what we thought we could do.
And artificial general intelligence?
When someone argues that artificial general intelligence will inevitably arrive because humanity invented aeroplanes, helicopters and computers, they are making precisely that error: turning past successes and the availability of resources into a guarantee of future success
This does not mean that artificial general intelligence is impossible. To claim that, we would first have to define rigorously what we mean by general intelligence, and then identify a logical, mathematical or physical limit that prevents its realisation. And to do this we need a philosophy that develops a definition on which we can agree, after answering many questions such as these: Is a system capable of solving problems in different disciplines necessarily intelligent? Does understanding mean processing information correctly? Is it possible to separate intelligence from consciousness, intentionality and subjective experience? will AGI be able to feel as though it were embodied? or will it be able to have equivalent sensations without having a body? is not having a body a requirement that can be waived? Will AGI have only training, or memories?
We have no general proof of the impossibility of AGI. But neither do we have a proof of its inevitability.
We can, of course, make predictions, invest, experiment and formulate hypotheses. What we certainly cannot do is turn a series of technological successes into a universal law according to which everything we imagine will become achievable if enough resources are devoted to it.
I believe that some of humanity’s most extraordinary intellectual achievements are precisely the proofs of its limits. Proving that something is impossible can represent an advance in human thought equal to that of an ingenious invention.