Their sample size was 16 people...

Great as an assistant for boring tasks. Still needs checking.

Can also help suggest improvements, but still needs checking.

Have to learn when to stop interacting with it and do it yourself.

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

Their sample size was 16 people...

Sounds reasonable. The time and energy ive lost on trying very confident chat gpt suggestions that doesnt work must be weeks at this point.

Sometimes its very good though and really helps, which is why its so frustrating. You never know if its going to work before you go through the process.

It has changed how me and coworkers work now also. We just talk to chat gpt instead of even trying to look something up in the docs and trying to understand it. Too slow to do that now, it feels like. There is a pressure to solve anything quickly now that chat gpt exists.

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

True and not true at the same time. Using agents indeed often don't work, mostly when I'm trying to do the wrong thing. Because then, AI agent does not say "the way you do it is overly complicated, it does not make any sense", but instead it says: "excellent idea, here are X steps I need to do to make it happen". It wasted my time many times, but it also guided me quickly though some problems that would take hours to research. Some of my projects wouldn't have been finished without AI.

True and not true at the same time. Using agents indeed often don't work, mostly when I'm trying to do the wrong thing. Because then, AI agent does not say "the way you do it is overly complicated, it does not make any sense", but instead it says: "excellent idea, here are X steps I need to do to make it happen". It wasted my time many times, but it also guided me quickly though some problems that would take hours to research. Some of my projects wouldn't have been finished without AI.

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

Their sample size was 16 people...

The main issue i have with AI coding, hasn't been the code. Its a bit ham fisted and overly naive, it is as if it's speed blind.

The main issue is that some of the code is out of date using functions that are deprecated etc, and it seems to be mixing paradigms and styles across languages in a very frustrating? way.

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

A "junior" project manager at my company vibe coded an entire full stack web app with one of those LLM IDEs. His background is industrial engineering and claims to have basically no programming experience.

It "works", as in, it does what it's meant to, but as you can guess, it relies on calls to LLM APIs where it really doesn't have to, and has several critical security flaws, inconsistencies in project structure and convention, and uses deprecated library features.

He already pitched it to one of our largest clients, and they're on board. They want to start testing at the end of the month.

He's had one junior dev who's been managing to keep things somewhat stable, but the poor dude really had his work cut out for him. I only recently joined the project because "it sounded cool", so I've been trying to fix some flaws while adding new requested features.

I've never worked with the frameworks and libraries before, so it's a good opportunity to upskill, but god damn I don't know if I want my name on this project.

A similar thing is happening with my brother at a different company. An executive vibe coded a web application, but this thing absolutely did not work.

My brother basically had one night to get it into a working state. He somehow (ritalin) managed to do it. The next day they presented it to one of their major clients. They really want it.

These AI dev tools absolutely have a direct negative impact on developer productivity, but they also have an indirect impact where non-devs use them and pass their Eldritch abominations to the actual devs to fix, extend and maintain.

Two years ago, I was worried about AI taking dev jobs, but now it feels like, to me, we'll need more human devs than ever in the long run.

Like, weren't these things supposed to exponentially get better? Like, cool, gh copilot can fuck up my project files now.

pho

faux

Yep I've got a working iOS app, a v.2 branched and on the way, with a ton of MapKit integrations. Unfortunately I'm getting depreciation errors and having to constantly remind the AI that it's using old code, showing it examples of new code, and then watching it forget as we keep talking.

Still, I have a working iOS app, which only took a few hours. When Jack Dorsey said he'd vibe coded his new app in a long weekend, I'm like, hey me too.

I don't doubt this is true. I've been playing with an A.I and some fairly simple python scripts and it's so tedious to get the A.I. to actually do something to the script correctly. Learning to prompt is a skill all it's own.

In my experience it's much more useful for doing things like in AWS like create a Cloudformation template or look through user permissions for excess privileges or setup a backup schedule, like at scale when you have lots of accounts and users, etc.

I like to think typos like that confirm my humanity

shhh don’t let the bots in on our secret

also now I’m hungry for phở

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.