I have a better LLM benchmark:

"I have a priest, a child and a bag of candy and I have to take them to the other side of the river. I can only take one person/thing at a time. In what order should I take them?"

Claude Sonnet 4 decided that it's inappropriate and refused to answer. When I explain that the constraint is not to leave child alone with candy he provided a solution that leaves the child alone with candy.

Grok would provide a solution that doesn't leave the child alone with a priest but wouldn't explain why.

ChatGPT would say that "The priest can't be left alone with the child (or vice versa) for moral or safety concerns." directly and then provide wrong solution.

But yeah, they will know how to play chess...

I did say that, because this isn't a pie chart situation, it's a Venn diagram situation.

For instance, AI art is 99% theft and 60% garbage. It's both because there's overlap.

Stolen and bad aren't opposites, why would this be a dichotomy?

Oh, I enjoy lots of great art! But do you think I watch every film? Listen to every band? There's tons of shit out there!

Do you really believe, of all the songs that are written every day, that less than a third are crap? Even Taylor Swift doesn't publish everything she does. Sometimes you work on something for weeks and then end up tossing it in the bin. More often, you work on something for 30 minutes before deciding "I'm gonna start over, try something different". The majority of art is crap, but then you keep the stuff you think works.

And what's that expression, "good artists copy, great artists steal". I mean, that's a bit satirical, but the fact is, everything is derivative to some degree. It's not that there aren't new ideas, it's just that our new ideas are based on older ones. We stand on the shoulders of giants (or at least, on the shoulders of some people who came before us).

All I was really saying, was that the accusation "2 parts copying, 1 part crap", well honestly that's par for the course, that's how humans work. (And we do some great work that way).

Next up, we asked a shoe to write a haiku but it was beaten by a 30 year old HaikuMaker.

I once spent 45 minutes trying to get ChatGPT to write a haiku. It couldn't do it. It explained what syllables were, and the rules for the syllables in a haiku, but it didn't understand it.

What you are describing has nothing to do with the tool. It’s dishonesty which is different.

The idea is that instead of commissioning the cow on the field, you go to the AI and ask it for that and it gives you a cow in the field. If you claim you made it, you are lying but that would be true even if you paid an artist and then claimed the same.

So with AI made art you’ll say “this art was made by an Ai” and no one will be confused as to who takes the credit, because it belongs to the algorithm.

Have you ever made art in your life? Because a big part of art is mimicking. Like 98% of it is mimicking. I draw, write and have dabbled in making music and playing instruments. You can’t learn these skills without mimicking. And most artists don’t ever do anything truly original, that’s a rarity and even when it happens you can trace the influences to other artists if you know how to look.

You could argue that AI has not developed its own style yet but that’s bullshit too imo because everyone knows the default AI art style when they see it, so that means that AI has a distinctive style. Is it unique? Maybe not, but neither is the art style of most artists or writers or even musicians.

I have a better LLM benchmark:

"I have a priest, a child and a bag of candy and I have to take them to the other side of the river. I can only take one person/thing at a time. In what order should I take them?"

Claude Sonnet 4 decided that it's inappropriate and refused to answer. When I explain that the constraint is not to leave child alone with candy he provided a solution that leaves the child alone with candy.

Grok would provide a solution that doesn't leave the child alone with a priest but wouldn't explain why.

ChatGPT would say that "The priest can't be left alone with the child (or vice versa) for moral or safety concerns." directly and then provide wrong solution.

But yeah, they will know how to play chess...

but... but.... reasoning models! AGI! Singularity!
Seriously, what you're saying is true, but it's not what OpenAI & Co are trying to peddle, so these experiments are a good way to call them out on their BS.

Tbf they don’t really claim that when you read the research, thats mostly media hype and ceo assholes spinning words.

Its good at lots specific tasks like rewriting emails and summarising gives text, short roleplay, boilerplate code. Some undiscovered uses.

Anthropic latest claims they would not hire their own ai because of how hard it failed at the test they give, They didnt do that expecting validation but to measure how far we are still off from ai doing meaningful full work.

I really want to see an LLM vs LLM chess match. It'll be messy as hell.

I thought CoPilot was just a rebagged ChatGPT anyway?

It's a silly experiment anyway, there are very good AI chess grandmasters but they were actually trained to play chess, not predict the next word in a text.

I thought CoPilot was just a rebagged ChatGPT anyway?

It's a silly experiment anyway, there are very good AI chess grandmasters but they were actually trained to play chess, not predict the next word in a text.

It is entirely disingenuous to just pretend that LLMs are not being widely promoted, marketed, and discussed as AGI, as a superintelligence that people are familiar with from SciFi shows/movies, that is vastly more capable and knowledgeable than basically any single human.

Yes, people who actually understand tech understand that LLMs are not AGI, that your metaphor of wrong tool wrong job is apt.

... But seemingly about +90% of humanity, including the people who own and profit from LLMs, including all the other business owners/managers who just want to lower their employee headcount ... do not understand this, that an LLM is actually basically an extremely advanced text autocorrect system, that frequently and confidently lies, spits out nonsense, hallucinates, etc.

If you think it isn't reasonable to continuously point out that LLMs are not superintelligences, then you likely live in a bubble of tech nerds who probably still think their jobs or retirement are secure.

They're not.

If corpos keep smashing """AI""" into basically every industry to replace as many workers as possible... the economy will collapse, as capitalism doesn't work without consumers who have jobs, and an avalanche of errors will cascade and snowball through every system that replaces humans with them...

...and even if those two things were not broadly true...

...the amount of literal power/energy, clean water and financial capital that is required to run the whole economy on these services is wildly unsustainable, both short term economically, and medium term ecologically.

That's true. But people pointing out that the whole attempt is absurd and senseless also reinforces the point that current AI isn't what companies tout it as.

then you likely live in a bubble of tech nerds

Well, we are on Lemmy...

I just asked ChatGPT too (your exact prompt there) and it did give me the correct solution.

1. Take the child over
2. Go back alone
3. Take the candy over
4. Bring the child back
5. Take the priest over
6. Go back alone
7. Take the child over again

It didn't comment on moral concerns, though it did applaud itself for keeping the priest and the child separated without elaborating on why.

For S&G, Just asked it to do one: