Well, not irrelevant. Lots of our world is trying to treat the LLM output as human-like output, so if human's are going to treat LLM output the same way they treat human generated content, then we have to characterize, for the people, how their expectations are broken in that context.

So as weird as it may seem to treat a stastical content extrapolation engine in the context of social science, there's a great deal of the reality and investment that wants to treat it as "person equivalent" output and so it must be studied in that context, if for no other reason to demonstrate to people that it should be considered "weird".

That definition seems a bit shaky. Trump & co. are mentally ill but they do have a minimum of intelligence.

Ah, well then, if he tells the bot to not hallucinate and validate output there's no reason to not trust the output. After all, you told the bot not to, and we all know that self regulation works without issue all of the time.

It gave me flashbacks when the Replit guy complained that the LLM deleted his data despite being told in all caps not to multiple times.

People really really don't understand how these things work...

It didn’t lie to you or gaslight you because those are things that a person with agency does. Someone who lies to you makes a decision to deceive you for whatever reason they have. Someone who gaslights you makes a decision to behave like the truth as you know it is wrong in order to discombobulate you and make you question your reality.

The only thing close to a decision that LLMs make is: what text can I generate that statistically looks similar to all the other text that I’ve been given. The only reason they answer questions is because in the training data they’ve been provided, questions are usually followed by answers.

It’s not apologizing you to, it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere - it has no ability to be sincere because it doesn’t have any thoughts.

There is no thinking. There are no decisions. The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are, and the more we fall into the trap of these AI marketers about how close we are to truly thinking machines.

About halfway through the article they quote a paper from 2023:

Similarly, another study from 2023 found LLMs “hallucinated,” or produced incorrect information, in 69 to 88 percent of legal queries.

The LLM space has been changing very quickly over the past few years. Yes, LLMs today still "hallucinate", but you're not doing anyone a service by reporting in 2025 the state of the field over 2 years before.

Nah every human is smarter than advanced autocomplete.

As any modern computer system, LLMs are much better and smarter than us at certain tasks while terrible at others. You could say that having good memory and communication skills is part of what defines an intelligent person. Not everyone has those abilities, but LLMs do.

My point is, there's nothing useful coming out of the arguments over the semantics of the word "intelligence".

The people who make them don't really understand how they work either. They know how to train them and how the software works, but they don't really know how it comes up with the answers it comes up with. They just do a ron of trial and error. Correlation is all they really have. Which of course is how a lot of medical science works too. So they have good company.

In my experience it can, but it has been pretty uncommon. But I also don't usually ask questions with only one answer.

The only thing close to a decision that LLMs make is

That's not true. An "if statement" is literally a decision tree.

The only reason they answer questions is because in the training data they’ve been provided

This is technically true for something like GPT-1. But it hasn't been true for the models trained in the last few years.

it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere

It has a large amount of system prompts that alter default behaviour in certain situations. Such as not giving the answer on how to make a bomb. I'm fairly certain there are catches in place to not be overly apologetic to minimize any reputation harm and to reduce potential "liability" issues.

And in that scenario, yes I'm being gaslite because a human told it to.

There is no thinking

Partially agree. There's no "thinking" in sentient or sapient sense. But there is thinking in the academic/literal definition sense.

There are no decisions

Absolutely false. The entire neural network is billions upon billions of decision trees.

The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are

I promise you I know very well what LLMs and other AI systems are. They aren't alive, they do not have human or sapient level of intelligence, and they don't feel. I've actually worked in the AI field for a decade. I've trained countless models. I'm quite familiar with them.

But "gaslighting" is a perfectly fine description of what I explained. The initial conditions were the same and the end result (me knowing the truth and getting irritated about it) were also the same.

The use of language like "unaware" when people are discussing LLMs drives me crazy. LLMs aren't "aware" of anything. They do not have a capacity for awareness in the first place.

People need to stop taking about them using terms that imply thought or consciousness, because it subtly feeds into the idea that they are capable of such.

I watched this entire video just so that I could have an informed opinion. First off, this feels like two very separate talks:

The first part is a decent breakdown of how artificial neural networks process information and store relational data about that information in a vast matrix of numerical weights that can later be used to perform some task. In the case of computer vision, those weights can be used to recognize objects in a picture or video streams, such as whether something is a hotdog or not.

As a side note, if you look up Hinton’s 2024 Nobel Peace Prize in Physics, you’ll see that he won based on his work on the foundations of these neural networks and specifically, their training. He’s definitely an expert on the nuts and bolts about how neural networks work and how to train them.

He then goes into linguistics and how language can be encoded in these neural networks, which is how large language models (LLMs) work… by breaking down words and phrases into tokens and then using the weights in these neural networks to encode how these words relate to each other. These connections are later used to generate other text output related to the text that is used as input. So far so good.

At that point he points out these foundational building blocks have been used to lead to where we are now, at least in a very general sense. He then has what I consider the pivotal slide of the entire talk, labeled Large Language Models, which you can see at 17:22. In particular he has two questions at the bottom of the slide that are most relevant:

• Are they genuinely intelligent?
• Or are they just a form of glorified auto-complete that uses statistical regularities to pastiche together pieces of text that were created by other people?

The problem is: he never answers these questions. He immediately moves on to his own theory about how language works using an analogy to LEGO bricks, and then completely disregards the work of linguists in understanding language, because what do those idiots know?

At this point he brings up The long term existential threat and I would argue the rest of this talk is now science fiction, because it presupposes that understanding the relationship between words is all that is necessary for AI to become superintelligent and therefore a threat to all of us.

Which goes back to the original problem in my opinion: LLMs are text generation machines. They use neural networks encoded as a matrix of weights that can be used to predict long strings of text based on other text. That’s it. You input some text, and it outputs other text based on that original text.

We know that different parts of the brain have different responsibilities. Some parts are used to generate language, other parts store memories, still other parts are used to make our bodies move or regulate autonomous processes like our heartbeat and blood pressure. Still other bits are used to process images from our eyes and other parts reason about spacial awareness, while others engage in emotional regulation and processing.

Saying that having a model for language means that we’ve built an artificial brain is like saying that because I built a round shape called a wheel means that I invented the modern automobile. It’s a small part of a larger whole, and although neural networks can be used to solve some very difficult problems, they’re only a specific tool that can be used to solve very specific tasks.

Although Geoffrey Hinton is an incredibly smart man who mathematically understands neural networks far better than I ever will, extrapolating that knowledge out to believing that a large language model has any kind of awareness or actual intelligence is absurd. It’s the underpants gnome economic theory, but instead of:

1. Collect underpants
2. ?
3. Profit!

It looks more like:

1. Use neural network training to construct large language models.
2. ?
3. Artificial general intelligence!

If LLMs were true artificial intelligence, then they would be learning at an increasing rate as we give them more capacity, leading to the singularity as their intelligence reaches hockey stick exponential growth. Instead, we’ve been throwing a growing amount resources at these LLMs for increasingly smaller returns. We’ve thrown a few extra tricks into the mix, like “reasoning”, but beyond that, I believe it’s clear that we’re headed towards a local maximum that is far enough away from intelligence that would be truly useful (and represent an actual existential threat), but in actuality only resembles what a human can output well enough to fool human decision makers into trusting them to solve problems that they are incapable of solving.

There goes middle management

Oh god I just figured it out.

It was never they are good at their tasks, faster, or more money efficient.

They are just confident to stupid people.

Christ, it's exactly the same failing upwards that produced the c suite. They've just automated the process.

Oh good, so that means we can just replace the C-suite with LLMs then, right? Right?

An AI won't need a Golden Parachute when they inevitably fuck it all up.

Okay fine, the LLM does not take into account in the context of its prompt that yada yada. Happy now word police, or do I need to pay a fine too? The real problem is people are replacing their brains with chatbots owned by the rich so soon their thoughts and by extension the truth will be owned by the rich, but go off pat yourself on the back because you preserved your holy sentience spook for another day.

People should understand that words like "unaware" or "overconfident" are not even applicable to these pieces of software. We might build intelligent machines in the future but if you know how these large language models work, it is obvious that it doesn't even make sense to talk about the awareness, intelligence, or confidence of such systems.

I find it so incredibly frustrating that we've gotten to the point where the "marketing guys" are not only in charge, but are believed without question, that what they say is true until proven otherwise.

"AI" becoming the colloquial term for LLMs and them being treated as a flawed intelligence instead of interesting generative constructs is purely in service of people selling them as such. And it's maddening. Because they're worthless for that purpose.

believing that a large language model has any kind of awareness or actual intelligence is absurd

I (as a person who works professionally in the area and tries to keep up with the current academic publications) happen to agree with you. But my credences are somewhat reduced after considering the points Hinton raises.

I think it is worth considering that there are a handful of academically active models of consciousness; some well-respected ones like the CTM are not at all inconsistent with Hinton's statements