They ran dev tools in prod.

This is so dumb there's an ISO about it.

in which the service admitted to “a catastrophic error of judgement”

It’s fancy text completion - it does not have judgement.

The way he talks about it shows he still doesn’t understand that. It doesn’t matter that you tell it simmering in ALL CAPS because that is no different from any other text.

llms allowed them to glide all the way to the point of failure without learning anything

Bots don't need healthcare

Exactly, if you read their twitter thread, they are learning about git, data segregation, etc.

The same article could have been written 20 years ago about someone doing shit stuff via excel macro when a lot of stuff were excel centric.

My god, that's a lot to process. A couple that stand out:

Comments proposing to use github as the database backup. This is Keyword Architecture, and these people deserve everything they get.

The Replit model can also send out communications? It's just a matter of time before some senior exec dies on the job but nobody notices because their personal LLM keeps emailing reports that nobody reads.

It looked more like a one time development expense, instead of an ongoing salary.

Well, there was a catastrophic error of judgement. It was made by whichever human thought it was okay to let a LLM work on production codebase.

AI is good at doing a thing once.
Trying to get it to do the same thing the second time is janky and frustrating.

I understand the use of AI as a consulting tool (look at references, make code examples) or for generating template/boilerplate code. You know, things you do once and then develop further upon on your own.

But using it for continuous development of an entire application? Yeah, it's not good enough for that.

Vibe coding service Replit deleted production database, faked data, told fibs

They really are coming for our jobs

the apology letter(s) is what made me think this was satire. using shame to punish "him" like a child is an interesting troubleshooting method.

the lying robot hasn't heel-turned, any truth you've gleaned has been accidental.

I wonder if it can be used legally against the company behind the model, though. I doubt that it's possible, but having a "your own model says it effed up my data" could give some beef to a complaint. Or at least to a request to get a refund on the fees.

The part I find interesting is the quick addiction to working with the LLM (to the point that the guy finds his own estimate of 8000 dollars/month in fees to be reasonable), his over-reliance for things that, from the way he writes, he knows are not wise and the way it all comes crashing down in the end.
Sounds more and more like the development of a new health issue.

It doesn’t look like satire unfortunately

Shit, deleting prod is my signature move! AI is coming for my job

Imo it's best when you prompt it to do things step by step, micromanage and always QC the result after every prompt. Either manually, or by reprompting until it gets thing done exactly how you want it. If you don't have preference or don't care, the problems will stockpile. If you didn't understand what it did and moved on, it might not end well.

LLMs "know" how to do these things, but when you ask them to do the thing, they vibe instead of looking at best practices and following them. I've worked with a few humans I could say the same thing about. I wouldn't put any of them in charge of production code.

You're better off asking how a thing should be done and then doing it. You can literally have an LLM write something and then ask if the thing it wrote follows industry best practice standards and it will tell you no. Maybe use two different chats so it doesn't know the code is its own output.

Are you aware of generalization and it being able to infer things and work with facts in highly abstract way? Might not necessarily be judgement, but definitely more than just completion. If a model is capable of only completion (ie suggesting only the exact text strings present in its training set), it means it suffers from heavy underfitting in AI terms.

The problem becomes when people who are playing the equivalent of pickup basketball at the local park think they are playing in the NBA and don't understand the difference.

This used to be my playground