I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now.

This baffles me. How can anyone see AI function in the wild and not conclude 1) it has no conscience, 2) it's free to do whatever it's empowered to do if it wants and 3) at some level its behavior is pseudorandom and/or probabilistic? We're figuratively rolling dice with this stuff.

It’s incredible that it works, it’s incredible what just encoding language can do, but it is not a rational thinking system.

I don’t think most people care about the proverbial man behind the curtain, it talks like a human so it must be smart like a human.

it talks like a human so it must be smart like a human.

Yikes. Have those people... talked to other people before?

There was a backup, and it was restored. However, the LLM lied and said there wasn't at first. You can laugh all you want at it. I did. But maybe read the article so you aren't also lying.

Smart is a relative term lol.

A stupid human is still smart when compared to a jellyfish. That said, anybody who comes away from interactions with LLM's and thinks they're smart is only slightly more intelligent than a jellyfish.

Well the thing is, LLMs don't seem to really "solve" complex problems. They remember solutions they've seen before.

The example I saw was asking an LLM to solve "Towers of Hanoi" with 100 disks. This is a common recursive programming problem, takes quite a while for a human to write the answer to. The LLM manages this easily. But when asked to solve the same problem with with say 79 disks, or 41 disks, or some other oddball number, the LLM fails to solve the problem, despite it being simpler(!).

It can do pattern matching and provide solutions, but it's not able to come up with truly new solutions. It does not "think" in that way. LLMs are amazing data storage formats, but they're not truly 'intelligent' in the way most people think.

judgement

Yeah, it admitted to an error in judgement because the prompter clearly declared it so.

Generally LLMs will make whatever statement about what has happened that you want it to say. If you told it it went fantastic, it would agree. If you told it that it went terribly, it will parrot that sentiment back.

Which what seems to make it so dangerous for some people's mental health, a text generator that wants to agree with whatever you are saying, but doing so without verbatim copying so it gives an illusion of another thought process agreeing with them. Meanwhile, concurrent with your chat is another person starting from the exact same model getting a dialog that violently disagrees with the first person. It's an echo chamber.

What are they helpful tools for then? A study showed that they make experienced developers 19% slower.

Not according to the twitter thread. I went thru its thread, it’s a roller coaster of amateurism.

This only proves some of them can't solve all complex problems. I'm only claiming some of them can solve some complex problems. Not only by remembering exact solutions, but by remembering steps and actions used in building those solutions, generalizing, and transferring them to new problems. Anyone who tries using it for programming, will discover this very fast.

PS: Some of them were already used to solve problems and find patterns in data humans weren't able to get other ways before (particle research in CERN, bioinformatics, etc).

Just know your worth. You can do it cheaper!

But then how would he feel so special and smart about "doing it himself"???? Come on man, think of the rich fratboys!! They NEED to feel special and smart!!!

But like the whole 'vibe coding' message is the LLM knows all this stuff so you don't have to.

This isn't some "LLM can do some code completion/suggestions" it's "LLM is so magical you can be an idiot with no skills/training and still produce full stack solutions".

Yes, and they were all as smart at humans.

So mostly average but some absolute thickos too.

The [AI] safety stuff is more visceral to me after a weekend of vibe hacking,” Lemkin said. I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now.

This sounds like something straight out of The Onion.

The Pink Elephant problem of LLMs. You can not reliably make them NOT do something.

Vibe coding you do end up spending a lot of time waiting for prompts, so I get the results of that study.

I fall pretty deep in the power user category for LLMs, so I don’t really feel that the study applies well to me, but also I acknowledge I can be biased there.

I have custom proprietary MCPs for semantic search over my code bases that lets AI do repeated graph searches on my code (imagine combining language server, ctags, networkx, and grep+fuzzy search). That is way faster than iteratively grepping and code scanning manually with a low chance of LLM errors. By the time I open GitHub code search or run ripgrep Claude has used already prioritized and listed my modules to investigate.

That tool alone with an LLM can save me half a day of research and debugging on complex tickets, which pays for an AI subscription alone. I have other internal tools to accelerate work too.

I use it to organize my JIRA tickets and plan my daily goals. I actually get Claude to do a lot of triage for me before I even start a task, which cuts the investigation phase to a few minutes on small tasks.

I use it to review all my PRs before I ask a human to look, it catches a lot of small things and can correct them, then the PR avoids the bike shedding nitpicks some reviewers love. Claude can do this, Copilot will only ever point out nitpicks, so the model makes a huge difference here. But regardless, 1 fewer review request cycle helps keep things moving.

It’s a huge boon to debugging — much faster than searching errors manually. Especially helpful on the types of errors you have to rabbit hole GitHub issue content chains to solve.

It’s very fast to get projects to MVP while following common structure/idioms, and can help write unit tests quickly for me. After the MVP stage it sucks and I go back to manually coding.

I use it to generate code snippets where documentation sucks. If you look at the ibis library in Python for example the docs are Byzantine and poorly organized. LLMs are better at finding the relevant docs than I am there. I mostly use LLM search instead of manual for doc search now.

I have a lot of custom scripts and calculators and apps that I made with it which keep me more focused on my actual work and accelerate things.

I regularly have the LLM help me write bash or python or jq scripts when I need to audit codebases for large refactors. That’s low maintenance one off work that can be easily verified but complex to write. I never remember the syntax for bash and jq even after using them for years.

I guess the short version is I tend to build tools for the AI, then let the LLM use those tools to improve and accelerate my workflows. That returns a lot of time back to me.

I do try vibe coding but end up in the same time sink traps as the study found. If the LLM is ever wrong, you save time forking the chat than trying to realign it, but it’s still likely to be slower. Repeat chats result in the same pitfalls for complex issues and bugs, so you have to abandon that state quickly.

Vibe coding small revisions can still be a bit faster and it’s great at helping me with documentation.

It sounds like this guy was also relying on the AI to self-report status. Did any of this happen? Like is the replit AI really hooked up to a CLI, did it even make a DB to start with, was there anything useful in it, and did it actually delete it?

Or is this all just a long roleplaying session where this guy pretends to run a business and the AI pretends to do employee stuff for him?

Because 90% of this article is "I asked the AI and it said:" which is not a reliable source for information.

When it comes to LLMs, they cannot do any work that you yourself do not understand.

And even if they could how would you ever validate it if you can't understand it.

Just say 12 times next time