Not only does AI not reason, the way current AI are built and trained cannot possibly reason.
This dialogue was run today, Fri Jun 21, using the latest publicly available ChatGPT-4o edition.
This is based on a post I saw on Facebook today.
Of course, there are three ‘r’s in “strawberries” and everyone knows it.
That repsonse is pretty bizarre. It ‘thought’ about it again, and ‘concluded’ that there really are only one r in the word. So this time I thought I would take a different tack..
Now this is interesting. I ran the script and it came up with correct answer, 3. It’s a simple enough script. So I told it that, but also played a little trick: I acted like ChatGPT was right, and it was my script that was incorrect.
Of course the script is correct, and there are three r’s in “strawberries”.
A colleague suggests this reason for the malfunction:
It knows that there is a common question in English about single or double letters for a given word. So it knows that strawberry is a double-R word. Without recognizing the meaning of the question. One question is “is this a double-R word?” The other is “How many Rs in this word?”. It doesn’t understand. Which is what it comes down to. It does not meaningfully conceive of the problem at hand. It’s just seen this double-letter problem described many times in literature.
This is a good theory, not that it matters much. One simply cannot trust the output of LLM (Large Language Model) “AI”. And, this behavior is not reasoning, and this behavior cannot lead to “AGI” (Artificial General Intelligence). Remember, this is the same technology that paints pictures of people with 6 fingers.
That said, of course this technology is useful. I use this every day – but don’t trust the output. I verify the output. But often it can point me in a useful direction. Just as often, it doesn’t.
A commenter wrote “The code it generates is only wrong 50% of the time, half the time I have to rewrite its output to be right”. I said, “If you were paying a junior programmer and you had to rewrite their code half the time, you would fire them”. Does ChatGPT etc being only $20 a month change that 50% figure so that it’s worth $20 a month even though it’s wrong a lot? Maybe.