I'm looking for an article showing that LLMs don't know how they work internally
Technology
80
Beiträge
32
Kommentatoren
5
Aufrufe
-
You can prove it’s not by doing some matrix multiplication and seeing its matrix multiplication. Much easier way to go about it
Yes, neural networks can be implemented with matrix operations. What does that have to do with proving or disproving the ability to reason? You didn't post a relevant or complete thought
Your comment is like saying an audio file isn't really music because it's just a series of numbers.
-
You're confusing the confirmation that the LLM cannot explain it's under-the-hood reasoning as text output, with a confirmation of not being able to reason at all. Anthropic is not claiming that it cannot reason. They actually find that it performs complex logic and behavior like planning ahead.
No, they really don’t. It’s a large language model. Input cues instruct it as to which weighted path through the matrix to take. Those paths are complex enough that the human mind can’t hold all the branches and weights at the same time. But there’s no planning going on; the model can’t backtrack a few steps, consider different outcomes and run a meta analysis. Other reasoning models can do that, but not language models; language models are complex predictive translators.
-
No, they really don’t. It’s a large language model. Input cues instruct it as to which weighted path through the matrix to take. Those paths are complex enough that the human mind can’t hold all the branches and weights at the same time. But there’s no planning going on; the model can’t backtrack a few steps, consider different outcomes and run a meta analysis. Other reasoning models can do that, but not language models; language models are complex predictive translators.
To write the second line, the model had to satisfy two constraints at the same time: the need to rhyme (with "grab it"), and the need to make sense (why did he grab the carrot?). Our guess was that Claude was writing word-by-word without much forethought until the end of the line, where it would make sure to pick a word that rhymes. We therefore expected to see a circuit with parallel paths, one for ensuring the final word made sense, and one for ensuring it rhymes.
Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, with these plans in mind, it writes a line to end with the planned word.
actually read the research? -
I've read that article. They used something they called an "MRI for AIs", and checked e.g. how an AI handled math questions, and then asked the AI how it came to that answer, and the pathways actually differed. While the AI talked about using a textbook answer, it actually did a different approach. That's what I remember of that article.
But yes, it exists, and it is science, not TicTok
Thank you. I found the article, linkin the OP
-
"Researchers" did a thing I did the first day I was actually able to ChatGPT and came to a conclusion that is in the disclaimers on the ChatGPT website. Can I get paid to do this kind of "research?" If you've even read a cursory article about how LLMs work you'd know that asking them what their reasoning is for anything doesn't work because the answer would just always be an explanation of how LLMs work generally.
Very arrogant answer. Good that you have intuition, but the article is serious, especially given how LLMs are used today. The link to it is in the OP now, but I guess you already know everything...
-
There was a study by Anthropic, the company behind Claude, that developed another AI that they used as a sort of "brain scanner" for the LLM, in the sense that allowed them to see sort of a model of how the LLM "internal process" worked
Yes, that's it. I added the link in the OP,
-
I agree. This is the exact problem I think people need to face with nural network AIs. They work the exact same way we do. Even if we analysed the human brain it would look like wires connected to wires with different resistances all over the place with some other chemical influences.
I think everyone forgets that nural networks were used in AI to replicate how animal brains work, and clearly if it worked for us to get smart then it should work for something synthetic. Well we've certainly answered that now.
Everyone being like "oh it's just a predictive model and it's all math and math can't be intelligent" are questioning exactly how their own brains work. We are just prediction machines, the brain releases dopamine when it correctly predicts things, it self learns from correctly assuming how things work. We modelled AI off of ourselves. And if we don't understand how we work, of course we're not gonna understand how it works.
Even if LLM "neurons" and their interconnections are modeled to the biological ones, LLMs aren't modeled on human brain, where a lot is not understood.
The first thing is that how the neurons are organized is completely different. Think about the cortex and the transformer.
Second is the learning process. Nowhere close.
The fact explained in the article about how we do math, through logical steps while LLMs use resemblance is a small but meaningful example. And it also shows that you can see how LLMs work, it's just very difficult
-
Yes, neural networks can be implemented with matrix operations. What does that have to do with proving or disproving the ability to reason? You didn't post a relevant or complete thought
Your comment is like saying an audio file isn't really music because it's just a series of numbers.
Improper comparison; an audio file isn’t the basic action on data, it is the data; the audio codec is the basic action on the data
“An LLM model isn’t really an LLM because it’s just a series of numbers”
But the action of turning the series of numbers into something of value (audio codec for an audio file, matrix math for an LLM) are actions that can be analyzed
And clearly matrix multiplication cannot reason any better than an audio codec algorithm. It’s matrix math, it’s cool we love matrix math. Really big matrix math is really cool and makes real sounding stuff. But it’s just matrix math, that’s how we know it can’t think
-
I agree. This is the exact problem I think people need to face with nural network AIs. They work the exact same way we do. Even if we analysed the human brain it would look like wires connected to wires with different resistances all over the place with some other chemical influences.
I think everyone forgets that nural networks were used in AI to replicate how animal brains work, and clearly if it worked for us to get smart then it should work for something synthetic. Well we've certainly answered that now.
Everyone being like "oh it's just a predictive model and it's all math and math can't be intelligent" are questioning exactly how their own brains work. We are just prediction machines, the brain releases dopamine when it correctly predicts things, it self learns from correctly assuming how things work. We modelled AI off of ourselves. And if we don't understand how we work, of course we're not gonna understand how it works.
LLMs among other things lack the whole neurotransmitter "live" regulation aspect and plasticity of the brain.
We are nowhere near a close representation of actual brains. LLMs to brains are like a horse carriage compared to modern cars. Yes they have four wheels and they move, and cars also need four wheels and move, but that is far from being close to each other.
-
Improper comparison; an audio file isn’t the basic action on data, it is the data; the audio codec is the basic action on the data
“An LLM model isn’t really an LLM because it’s just a series of numbers”
But the action of turning the series of numbers into something of value (audio codec for an audio file, matrix math for an LLM) are actions that can be analyzed
And clearly matrix multiplication cannot reason any better than an audio codec algorithm. It’s matrix math, it’s cool we love matrix math. Really big matrix math is really cool and makes real sounding stuff. But it’s just matrix math, that’s how we know it can’t think
Do LLMs not exhibit emergent behaviour? But who am I, a simple skin-bag of chemicals, to really say.
-
LLMs among other things lack the whole neurotransmitter "live" regulation aspect and plasticity of the brain.
We are nowhere near a close representation of actual brains. LLMs to brains are like a horse carriage compared to modern cars. Yes they have four wheels and they move, and cars also need four wheels and move, but that is far from being close to each other.
So LLMs are like a human with anterograde amnesia. They're like Dory.
-
I agree. This is the exact problem I think people need to face with nural network AIs. They work the exact same way we do. Even if we analysed the human brain it would look like wires connected to wires with different resistances all over the place with some other chemical influences.
I think everyone forgets that nural networks were used in AI to replicate how animal brains work, and clearly if it worked for us to get smart then it should work for something synthetic. Well we've certainly answered that now.
Everyone being like "oh it's just a predictive model and it's all math and math can't be intelligent" are questioning exactly how their own brains work. We are just prediction machines, the brain releases dopamine when it correctly predicts things, it self learns from correctly assuming how things work. We modelled AI off of ourselves. And if we don't understand how we work, of course we're not gonna understand how it works.
They work the exact same way we do.
Two things being difficult to understand does not mean that they are the exact same.
-
You can prove it’s not by doing some matrix multiplication and seeing its matrix multiplication. Much easier way to go about it
People that can not do Matrix multiplication do not possess the basic concepts of intelligence now? Or is software that can do matrix multiplication intelligent?
-
Here's a book for a different audience. Explains in layman terms why to be wary about this tech, https://thebullshitmachines.com/
-
I found the aeticle in a post on the fediverse, and I can't find it anymore.
The reaserchers asked a simple mathematical question to an LLM ( like 7+4) and then could see how internally it worked by finding similar paths, but nothing like performing mathematical reasoning, even if the final answer was correct.
Then they asked the LLM to explain how it found the result, what was it's internal reasoning. The answer was detailed step by step mathematical logic, like a human explaining how to perform an addition.
This showed 2 things:
-
LLM don't "know" how they work
-
the second answer was a rephrasing of original text used for training that explain how math works, so LLM just used that as an explanation
I think it was a very interesting an meaningful analysis
Can anyone help me find this?
EDIT: thanks to @theunknownmuncher
@lemmy.world
https://www.anthropic.com/research/tracing-thoughts-language-model its this oneEDIT2: I'm aware LLM dont "know" anything and don't reason, and it's exactly why I wanted to find the article. Some more details here: https://feddit.it/post/18191686/13815095
It's the anthropic article you are looking for, where they performed open brain surgery equivalent to find out that they do maths in very strange and eerily humanlike operations, like they will estimate, then if it goes over calculate the last digit like I do. It sucks as a counting technique though
-
-
Improper comparison; an audio file isn’t the basic action on data, it is the data; the audio codec is the basic action on the data
“An LLM model isn’t really an LLM because it’s just a series of numbers”
But the action of turning the series of numbers into something of value (audio codec for an audio file, matrix math for an LLM) are actions that can be analyzed
And clearly matrix multiplication cannot reason any better than an audio codec algorithm. It’s matrix math, it’s cool we love matrix math. Really big matrix math is really cool and makes real sounding stuff. But it’s just matrix math, that’s how we know it can’t think
LOL you didn't really make the point you thought you did. It isn't an "improper comparison" (it's called a false equivalency FYI), because there isn't a real distinction between information and this thing you just made up called "basic action on data", but anyway have it your way:
Your comment is still exactly like saying an audio pipeline isn't really playing music because it's actually just doing basic math.
-
I agree. This is the exact problem I think people need to face with nural network AIs. They work the exact same way we do. Even if we analysed the human brain it would look like wires connected to wires with different resistances all over the place with some other chemical influences.
I think everyone forgets that nural networks were used in AI to replicate how animal brains work, and clearly if it worked for us to get smart then it should work for something synthetic. Well we've certainly answered that now.
Everyone being like "oh it's just a predictive model and it's all math and math can't be intelligent" are questioning exactly how their own brains work. We are just prediction machines, the brain releases dopamine when it correctly predicts things, it self learns from correctly assuming how things work. We modelled AI off of ourselves. And if we don't understand how we work, of course we're not gonna understand how it works.
I agree. This is the exact problem I think people need to face with nural network AIs. They work the exact same way we do.
I don't think this is a fair way of summarizing it. You're making it sound like we have AGI, which we do not have AGI and we may never have AGI.
-
Even if LLM "neurons" and their interconnections are modeled to the biological ones, LLMs aren't modeled on human brain, where a lot is not understood.
The first thing is that how the neurons are organized is completely different. Think about the cortex and the transformer.
Second is the learning process. Nowhere close.
The fact explained in the article about how we do math, through logical steps while LLMs use resemblance is a small but meaningful example. And it also shows that you can see how LLMs work, it's just very difficult
I agree, but I'm not sure it matters when it comes to the big questions, like "what separates us from the LLMs?" Answering that basically amounts to answering "what does it mean to be human?", which has been stumping philosophers for millennia.
It's true that artificial neurons are significant different than biological ones, but are biological neurons what make us human? I'd argue no. Animals have neurons, so are they human? Also, if we ever did create a brain simulation that perfectly replicated someone's brain down to the cellular level, and that simulation behaved exactly like the original, I would characterize that as a human.
It's also true LLMs can't learn, but there are plenty of people with anterograde amnesia that can't either.
This feels similar to the debates about what separates us from other animal species. It used to be thought that humans were qualitatively different than other species by virtue of our use of tools, language, and culture. Then it was discovered that plenty of other animals use tools, have language, and something resembling a culture. These discoveries were ridiculed by many throughout the 20th century, even by scientists, because they wanted to keep believing humans are special in some qualitative way. I see the same thing happening with LLMs.
-
Improper comparison; an audio file isn’t the basic action on data, it is the data; the audio codec is the basic action on the data
“An LLM model isn’t really an LLM because it’s just a series of numbers”
But the action of turning the series of numbers into something of value (audio codec for an audio file, matrix math for an LLM) are actions that can be analyzed
And clearly matrix multiplication cannot reason any better than an audio codec algorithm. It’s matrix math, it’s cool we love matrix math. Really big matrix math is really cool and makes real sounding stuff. But it’s just matrix math, that’s how we know it can’t think
Can humans think?
-
More than enough people who claim to know how it works think it might be "evolving" into a sentient being inside it's little black box. Example from a conversation I gave up on...
https://sh.itjust.works/comment/18759960Maybe I should rephrase my question:
Outside of comment sections on the internet, who has claimed or is claiming that LLMs have the capacity to reason?
