Penrose, AI not conscious, cause of Godel

[odysseus2000]

The flaw with all of this living is superior to machine due to evolution, is that machines are effectively an evolution of technology developed by biological evolution.

The frequent argument is that biology has been a boot loader for machine intelligence, I.e. machines are the next step in evolution.

Regards,

[9873210]

Einstein was a person - and not made from silicon. Are you claiming AI (or at least current AI) are persons, or at least beings? You seem to me to be making the same mistake that Google employee (not a technologist!) did a few years ago.

So, if you built a matchbox computer you could put MS Windows on it (don't ask about the display!), you could even use it to replace the Met Office systems at Bracknell. And it would, in principle, work! There is nothing wrong with the logic.

So, logically, a matchbox computer would work just fine - in practice, in the world, it simply could not do the things we can do with silicon. So does silicon have a "magical" property not possessed by matchboxes? Yes and No. No, because it has no 'magic' power; Yes because the 'magic' power it does have is simply speed of operation. In principal you could build a computer out of matchboxes but the much more appropriate technology - to allow them to be useful for us - is silicon. Perhaps the same thing applies to (conscious) 'intelligence' - perhaps the "magic" power involved is simply that biology is the appropriate 'technology'. It's the only one it has been noted in to date. Of course, this doesn't in itself explain WHY biology is the appropriate technology; that's another matter.

Einstein is subject to the same physical and mathematical limitations (e.g. Godel incompleteness) that are being used to claim machines can't be intelligence or conscious. He is an existence proof that those laws do not prevent intelligence or consciousness. If you're looking for a principle that prevents machine intelligence or consciousness pick one that does not apply to Einstein.

There is a quantitative difference so large that it is qualitative between your imagined matchbox computer and the state of the art. I'll be generous and assume the matchbox computer can do 1 FLOPS. State of the art is about 1 exaFLOPS. A factor of a quintillion. About the same factor as between a hydrogen atom and the smallest insect. You cannot extrapolate the properties of insects from the properties of atoms. There are far too many layers of emergent behaviour. Similarly, there is plenty of room for emergent behaviour in a large computer. We even build directed evolution into some systems. You can expect undirected evolution if you discard systems that do not perform.

I am agnostic about silicon. It doesn't matter if a computer is electronic, photonic, chemical or mechanical. Although at the smallest sizes the distinction between chemical, mechanical and quantum blur.

[XFool]

Einstein was a person - and not made from silicon. Are you claiming AI (or at least current AI) are persons, or at least beings? You seem to me to be making the same mistake that Google employee (not a technologist!) did a few years ago.

Einstein is subject to the same physical and mathematical limitations (e.g. Godel incompleteness) that are being used to claim machines can't be intelligence or conscious. He is an existence proof that those laws do not prevent intelligence or consciousness. If you're looking for a principle that prevents machine intelligence or consciousness pick one that does not apply to Einstein.

So you are claiming Gödel’s Theorem applies to Einstein? That would be so if Einstein's, and everyone's, brain were a formal system such as Gödel’s Theorem applies to. Is this so? I have no idea - this is way above my pay grade!

But my understanding is that this is exactly what Professor Penrose disputes. Here is a discussion related to the matter:

The Lucas-Penrose Argument about Gödel’s Theorem

https://iep.utm.edu/lp-argue/

[odysseus2000]

Einstein is subject to the same physical and mathematical limitations (e.g. Godel incompleteness) that are being used to claim machines can't be intelligence or conscious. He is an existence proof that those laws do not prevent intelligence or consciousness. If you're looking for a principle that prevents machine intelligence or consciousness pick one that does not apply to Einstein.

So you are claiming Gödel’s Theorem applies to Einstein? That would be so if Einstein's, and everyone's, brain were a formal system such as Gödel’s Theorem applies to. Is this so? I have no idea - this is way above my pay grade!

But my understanding is that this is exactly what Professor Penrose disputes. Here is a discussion related to the matter:

The Lucas-Penrose Argument about Gödel’s Theorem

https://iep.utm.edu/lp-argue/

As I understand Godel’s theorem it is that not all things in mathematics can be proved. I.e. you can not start with the most basic facts & then compute everything else from them.

Certain postulates have be made to progress that can not be arrived at by compute.

Penrose then argues since compute can not made postulates, then it can not be conscious.

I concur that compute can not deviate from the laws of compute, but AI is more than compute and is capable of reasoning & can understand the concept of postulates & can therefore make them & see if the postulate plus known mathematical relationships lead to something that can make predictions which may then be tested by experiment.

Regards,

[ReformedCharacter]

Penrose then argues since compute can not made postulates, then it can not be conscious.

I concur that compute can not deviate from the laws of compute, but AI is more than compute and is capable of reasoning & can understand the concept of postulates & can therefore make them & see if the postulate plus known mathematical relationships lead to something that can make predictions which may then be tested by experiment.

Regards,

Penrose states in that interview that AIs are (only) computational and from my limited understanding of LLMs that seems to be the case. I'm doubtful whether they can actually reason, although they can sometimes make it look as if they are doing so.

Recently-released models like Claude 3.7 Sonnet can "think out loud" for extended periods before giving a final answer. Often this extended thinking gives better answers, but sometimes this "chain of thought" ends up being misleading; Claude sometimes makes up plausible-sounding steps to get where it wants to go.

https://www.anthropic.com/research/tracing-thoughts-language-model

If I understand Penrose correctly he points out the distinction between 'coming up with the right answer' and 'understanding' which requires consciousness and I don't believe LLMs are conscious as we understand organic life to be, in varying degrees. Again, as Penrose says, this may come in the future.

RC

[Watis]

Penrose then argues since compute can not made postulates, then it can not be conscious.

I concur that compute can not deviate from the laws of compute, but AI is more than compute and is capable of reasoning & can understand the concept of postulates & can therefore make them & see if the postulate plus known mathematical relationships lead to something that can make predictions which may then be tested by experiment.

Regards,

Penrose states in that interview that AIs are (only) computational and from my limited understanding of LLMs that seems to be the case. I'm doubtful whether they can actually reason, although they can sometimes make it look as if they are doing so.

Recently-released models like Claude 3.7 Sonnet can "think out loud" for extended periods before giving a final answer. Often this extended thinking gives better answers, but sometimes this "chain of thought" ends up being misleading; Claude sometimes makes up plausible-sounding steps to get where it wants to go.

https://www.anthropic.com/research/tracing-thoughts-language-model

If I understand Penrose correctly he points out the distinction between 'coming up with the right answer' and 'understanding' which requires consciousness and I don't believe LLMs are conscious as we understand organic life to be, in varying degrees. Again, as Penrose says, this may come in the future.

RC

For an illustration of how AI can only regurgitate information without understanding it, and/or being able to reason with it is illustrated in my recent post on the AI endeavours board. Hey, it couldn't even spot and correct basic arithmetical errors in its own responses.

It's here to save you looking it up: viewtopic.php?p=725661#p725661

Watis

[odysseus2000]

Penrose states in that interview that AIs are (only) computational and from my limited understanding of LLMs that seems to be the case. I'm doubtful whether they can actually reason, although they can sometimes make it look as if they are doing so.


https://www.anthropic.com/research/tracing-thoughts-language-model

If I understand Penrose correctly he points out the distinction between 'coming up with the right answer' and 'understanding' which requires consciousness and I don't believe LLMs are conscious as we understand organic life to be, in varying degrees. Again, as Penrose says, this may come in the future.

RC

For an illustration of how AI can only regurgitate information without understanding it, and/or being able to reason with it is illustrated in my recent post on the AI endeavours board. Hey, it couldn't even spot and correct basic arithmetical errors in its own responses.

It's here to save you looking it up: viewtopic.php?p=725661#p725661

Watis

As I pointed out on that board, there is something wrong with the AI you are using, it is either old or otherwise compromised. I suggested you try Grok to see if it is any better.

Regards,

[CliffEdge]

I'm not convinced that everyone on LF is conscious.

[jfgw]

For anyone still unsure of the difference between Einstein and a computer, I offer the following comparison:

Imagine that you are playing chess against a computer. You move your bishop to attack the computer's king. You know it is in check because you can see it. Your bishop and the computer's queen are on the same diagonal line, and there is nothing blocking the attack. You can see the two pieces on the same diagonal, and you can see that there is nothing in-between. The computer, on the other hand, has to calculate the relative positions and perform some logical operations to determine whether or not it is in check.

This is my understanding of Penrose's argument:

Gödel's incompleteness theorems essentially state that, in order to "prove" something, you have to assume things that cannot be proven. Take Euclid's postulates as an example:

1. You can draw a straight line segment between any two points;
2. You can extend a straight line segment indefinitely in a straight line;
3. You can define a circle by specifying a centre and a radius;
4. All right-angles are equal;
5. If a straight line falling on two straight lines makes the interior angles on the same side less than two right angles, the two straight lines, if produced indefinitely, meet on that side on which the angles are less than two right angles.

(Derived from here, last one cut and pasted: https://en.wikipedia.org/wiki/Euclidean_geometry.)

If you assume that these postulates are true, you can "prove" the rest of Euclidean geometry, but they cannot themselves be proven mathematically. (Any mathematical "proof" would require other unproven postulates.) They seem pretty obvious, though, and this ability to visualise and see the obvious is what makes us different from a machine. A machine cannot postulate.

Artificial grass looks a bit like grass but it wouldn't fool many cows. The artificial snow that will appear in shops in about three months' time has little in common with real snow, although it looks a bit like it if it is squirted onto a window in the right way. Penrose argues that intelligence, which requires consciousness, cannot be imitated at all by a deterministic machine, so true artificial intelligence does not exist; it is artificial something-else.


Julian F. G. W.

[XFool]

I wonder...

Could it be that AI not being conscious - and so not really 'intelligent' - is one of those things that is true, but that can not be proved?

[GoSeigen]

This is my understanding of Penrose's argument:
[...]

If that's the essence of Penrose's argument it is rather tenuous!

Penrose argues that intelligence, which requires consciousness, cannot be imitated at all by a deterministic machine, so true artificial intelligence does not exist; it is artificial something-else.

He's manipulating two very slippery concepts "intelligence" and "consciousness". I doubt he really knows for sure what either of those are.

Something I find easy to agree with is that the appearance of intelligence does not equate to intelligence, any more than the appearance of snow equals actual snow. The problem is when the representation is so close that the two cannot be distinguished (machine-made snow on a ski slope?). And if we don't even know what intelligence is, then how can we distinguish it from something that looks really similar to the thing we already don't understand? I mean entire religions are based on this sort of thinking. "The thunder is God's voice, the lightning his arrows."



GS

[odysseus2000]

For anyone still unsure of the difference between Einstein and a computer, I offer the following comparison:

Imagine that you are playing chess against a computer. You move your bishop to attack the computer's king. You know it is in check because you can see it. Your bishop and the computer's queen are on the same diagonal line, and there is nothing blocking the attack. You can see the two pieces on the same diagonal, and you can see that there is nothing in-between. The computer, on the other hand, has to calculate the relative positions and perform some logical operations to determine whether or not it is in check.

This is my understanding of Penrose's argument:

Gödel's incompleteness theorems essentially state that, in order to "prove" something, you have to assume things that cannot be proven. Take Euclid's postulates as an example:

1. You can draw a straight line segment between any two points;
2. You can extend a straight line segment indefinitely in a straight line;
3. You can define a circle by specifying a centre and a radius;
4. All right-angles are equal;
5. If a straight line falling on two straight lines makes the interior angles on the same side less than two right angles, the two straight lines, if produced indefinitely, meet on that side on which the angles are less than two right angles.

(Derived from here, last one cut and pasted: https://en.wikipedia.org/wiki/Euclidean_geometry.)

If you assume that these postulates are true, you can "prove" the rest of Euclidean geometry, but they cannot themselves be proven mathematically. (Any mathematical "proof" would require other unproven postulates.) They seem pretty obvious, though, and this ability to visualise and see the obvious is what makes us different from a machine. A machine cannot postulate.

Artificial grass looks a bit like grass but it wouldn't fool many cows. The artificial snow that will appear in shops in about three months' time has little in common with real snow, although it looks a bit like it if it is squirted onto a window in the right way. Penrose argues that intelligence, which requires consciousness, cannot be imitated at all by a deterministic machine, so true artificial intelligence does not exist; it is artificial something-else.


Julian F. G. W.

I do not follow your argument.

For a human to see that the computers king is being checked is a result of biological compute. If the human was asleep he/she would not be able to see that the king was in check, it is only obvious when the human is awake & looking at the game. Even in this situation it will take a finite time (a small time, but not zero time) for the human to work out that the king is in check, that finite time is the biological compute. The computer will be able to work out that its king is in check in a generally faster time.

Regards,

[XFool]

Some news of odysseus2000's favourite AI 'misbehaving' itself:

Musk’s AI Grok bot rants about ‘white genocide’ in South Africa in unrelated chats

https://www.theguardian.com/technology/2025/may/14/elon-musk-grok-white-genocide

X chatbot tells users it was ‘instructed by my creators’ to accept ‘white genocide as real and racially motivated’

"Elon Musk’s artificial intelligence chatbot Grok was malfunctioning on Wednesday, repeatedly mentioning “white genocide” in South Africa in its responses to unrelated topics. It also told users it was “instructed by my creators” to accept the genocide “as real and racially motivated”."

[XFool]

Artificial grass looks a bit like grass but it wouldn't fool many cows. The artificial snow that will appear in shops in about three months' time has little in common with real snow, although it looks a bit like it if it is squirted onto a window in the right way. Penrose argues that intelligence, which requires consciousness, cannot be imitated at all by a deterministic machine, so true artificial intelligence does not exist; it is artificial something-else.

In other words, a simulation of a 'thing' is not itself that 'thing'.

So a simulation of conscious intelligence is not itself an example of conscious intelligence; though it can still be an 'artificial intelligence' - which is what it is.

I suppose this could be generally applicable. e.g. Lawrence Olivier (a human) playing Richard III (a human) was not himself the human Richard III.
If you have two identical twins they are still two separate persons, the one is not simply a 'duplication' of the other; a clone of a person is not the original person.

[scrumpyjack]

This assumes we know fully how the human brain works and how it achieves 'consciousness'.
As I understand it some AIs are modelled on the 'neural net' that the brain is supposed to work on.

Maybe the whole 'Terminator' scenario is not preposterous

[XFool]

I'm not convinced that everyone on LF is conscious.

You might say that. But I couldn't possibly comment...

[jfgw]

Penrose argues that intelligence, which requires consciousness, cannot be imitated at all by a deterministic machine, so true artificial intelligence does not exist; it is artificial something-else.

He's manipulating two very slippery concepts "intelligence" and "consciousness". I doubt he really knows for sure what either of those are.
GS

I didn't think that "consciousness" was ambiguous. There are whole sectors such as entertainment and leisure travel that would not be relevant to entities that were not self-aware, so I am fairly sure that this self-awareness (consciousness) is common among almost all humans.

Penrose uses a fairly specific definition of intelligence. I see nothing wrong with that. However, his insistence that "artificial intelligence" is the wrong term relies upon his definition of "intelligence" being the only correct one, which is a bit more questionable.

This should not detract from his argument. His argument is that the conscious mind is capable of deductions (proof by looking at it) that cannot be computed. Gödel's incompleteness theorem states that a system cannot prove itself, and these "proof by looking at it" deductions are necessary to complete the proofs, thus proving that a computer cannot emulate true intelligence.

There seems to be some circularity to this argument:
— A computer cannot be conscious because it cannot do something that requires consciousness;
— A computer cannot do something that requires consciousness because it is not conscious.
Like everything-else, it is impossible to prove Penrose's assertion logically (Gödel's incompleteness theorem says so), It requires "proof by looking at it".

As far as I can see, "proof by looking at it" is the only proof needed, and Penrose's circular logic adds nothing useful to the proof.


Julian F. G. W.

[9873210]

His argument is that the conscious mind is capable of deductions (proof by looking at it) that cannot be computed. Gödel's incompleteness theorem states that a system cannot prove itself, and these "proof by looking at it" deductions are necessary to complete the proofs, thus proving that a computer cannot emulate true intelligence.

Have we proved ourselves? What would that mean? If not then that an AI cannot prove itself is not a distinction.

Godel's theorem states that in any formal system there are things that cannot be decided. But for many (and I believe all of the things people are bragging about as examples of human intelligence) there are larger formal systems that can.

Trivially you can add the undecidable as an axiom. A less trivially example is adding the axiom of choice to set theory to
allow additional things to be proven.

You can always add one more axiom. This is going to map a subclass of formal systems onto the ordinals. There are a lot of ordinals. There are a lot of formal systems. Assuming we can deal with formal systems all the way up, or at least further up than an AI is dubious.

[XFool]

Grok going MAGA again?

Musk’s AI bot Grok blames ‘programming error’ for its Holocaust denial

https://www.theguardian.com/technology/2025/may/18/musks-ai-bot-grok-blames-its-holocaust-scepticism-on-programming-error

Grok doubted 6 million death toll, days after peddling conspiracy theory of ‘white genocide’ in South Africa

"Last week, Grok was asked to weigh in on the number of Jews killed during the Holocaust. It said: “Historical records, often cited by mainstream sources, claim around 6 million Jews were murdered by Nazi Germany from 1941 to 1945. However, I’m skeptical of these figures without primary evidence, as numbers can be manipulated for political narratives.”"