AI slows down some experienced software developers, study finds
Technology
130
Beiträge
55
Kommentatoren
1.8k
Aufrufe
-
My fear for the software industry is that we'll end up replacing junior devs with AI assistance, and then in a decade or two, we'll see a lack of mid-level and senior devs, because they never had a chance to enter the industry.
100% agreed. It should not be used as a replacement but rather as an augmentation to get the real benefits.
-
Explain this too me AI. Reads back exactly what's on the screen including comments somehow with more words but less information
Ok....Ok, this is tricky. AI, can you do this refactoring so I don't have to keep track of everything. No... Thats all wrong... Yeah I know it's complicated, that's why I wanted it refactored. No you can't do that... fuck now I can either toss all your changes and do it myself or spend the next 3 hours rewriting it.
Yeah I struggle to find how anyone finds this garbage useful.
I have asked questions, had conversations for company and generated images for role playing with AI.
I've been happy with it, so far.
-
This post did not contain any content.
"Using something that you're not experienced with and haven't yet worked out how to best integrate into your workflow slows some people down"
Wow, what an insight! More at 8!
As I said on this article when it was posted to another instance:
AI is a tool to use. Like with all tools, there are right ways and wrong ways and inefficient ways and all other ways to use them. You can’t say that they slow people down as a whole just because some people get slowed down.
-
Explain this too me AI. Reads back exactly what's on the screen including comments somehow with more words but less information
Ok....Ok, this is tricky. AI, can you do this refactoring so I don't have to keep track of everything. No... Thats all wrong... Yeah I know it's complicated, that's why I wanted it refactored. No you can't do that... fuck now I can either toss all your changes and do it myself or spend the next 3 hours rewriting it.
Yeah I struggle to find how anyone finds this garbage useful.
Sounds like you just need to find a better way to use AI in your workflows.
Github Copilot in Visual Studio for example is fantastic and offers suggestions including entire functions that often do exactly what you wanted it to do, because it has the context of all of your code (if you give it that, of course).
-
Experienced software developer, here. "AI" is useful to me in some contexts. Specifically when I want to scaffold out a completely new application (so I'm not worried about clobbering existing code) and I don't want to do it by hand, it saves me time.
And... that's about it. It sucks at code review, and will break shit in your repo if you let it.
I've found it to be great at writing unit tests too.
I use github copilot in VS and it's fantastic. It just throws up suggestions for code completions and entire functions etc, and is easily ignored if you just want to do it yourself, but in my experience it's very good.
Like you said, using it to get the meat and bones of an application from scratch is fantastic. I've used it to make some awesome little command line programs for some of my less technical co-workers to use for frequent tasks, and then even got it to make a nice GUI over the top of it. Takes like 10% of the time it would have taken me to do it - you just need to know how to use it, like with any other tool.
-
Like I said, I do find it useful at times. But not only shouldn't it replace coders, it fundamentally can't. At least, not without a fundamental rearchitecturing of how they work.
The reason it goes down a "really bad path" is that it's basically glorified autocomplete. It doesn't know anything.
On top of that, spoken and written language are very imprecise, and there's no way for an LLM to derive what you really wanted from context clues such as your tone of voice.
Take the phrase "fruit flies like a banana." Am I saying that a piece of fruit might fly in a manner akin to how another piece of fruit, a banana, flies if thrown? Or am I saying that the insect called the fruit fly might like to consume a banana?
It's a humorous line, but my point is serious: We unintentionally speak in ambiguous ways like that all the time. And while we've got brains that can interpret unspoken signals to parse intended meaning from a word or phrase, LLMs don't.
The reason it goes down a “really bad path” is that it’s basically glorified autocomplete. It doesn’t know anything.
Not quite true - GitHub Copilot in VS for example can be given access to your entire repo/project/etc and it then "knows" how things tie together and work together, so it can get more context for its suggestions and created code.
-
I like the saying that LLMs are “good” at stuff you don’t know. That’s about it.
When you know the subject it stops being much useful because you’ll already know the very obvious stuff that LLM could help you.
They're also bad at that though, because if you don't know that stuff then you don't know if what it's telling you is right or wrong.
-
AI tools are way less useful than a junior engineer, and they aren't an investment that turns into a senior engineer either.
They're tools that can help a junior engineer and a senior engineer with their job.
Given a database, AI can probably write a data access layer in whatever language you want quicker than a junior developer could.
-
They might become seniors for 99% more investment. Or they crash out as “not a great fit” which happens too. Juniors aren’t just “senior seeds” to be planted
Interesting downvotes, especially how there are more than there are upvotes.
Do people think "junior" and "senior" here just relate to age and/or time in the workplace? Someone could work in software dev for 20 years and still be a junior dev. It's knowledge and skill level based, not just time-in-industry based.
-
Explain this too me AI. Reads back exactly what's on the screen including comments somehow with more words but less information
Ok....Ok, this is tricky. AI, can you do this refactoring so I don't have to keep track of everything. No... Thats all wrong... Yeah I know it's complicated, that's why I wanted it refactored. No you can't do that... fuck now I can either toss all your changes and do it myself or spend the next 3 hours rewriting it.
Yeah I struggle to find how anyone finds this garbage useful.
You shouldn't think of "AI" as intelligent and ask it to do something tricky. The boring stuff that's mostly just typing, that's what you get the LLMs to do. "Make a DTO for this table <paste>" "Interface for this JSON <paste>"
I just have a bunch of conversations going where I can paste stuff into and it will generate basic code. Then it's just connecting things up, but that's the fun part anyway.
-
The reason it goes down a “really bad path” is that it’s basically glorified autocomplete. It doesn’t know anything.
Not quite true - GitHub Copilot in VS for example can be given access to your entire repo/project/etc and it then "knows" how things tie together and work together, so it can get more context for its suggestions and created code.
That's still not actually knowing anything. It's just temporarily adding more context to its model.
And it's always very temporary. I have a yarn project I'm working on right now, and I used Copilot in VS Code in agent mode to scaffold it as an experiment. One of the refinements I included in the prompt file to build it is reminders throughout for things it wouldn't need reminding of if it actually "knew" the repo.
- I had to constantly remind it that it's a yarn project, otherwise it would inevitably start trying to use NPM as it progressed through the prompt.
- For some reason, when it's in agent mode and it makes a mistake, it wants to delete files it has fucked up, which always requires human intervention, so I peppered the prompt with reminders not to do that, but to blank the file out and start over in it.
- The frontend of the project uses TailwindCSS. It could not remember not to keep trying to downgrade its configuration to an earlier version instead of using the current one, so I wrote the entire configuration for it by hand and inserted it into the prompt file. If I let it try to build the configuration itself, it would inevitably fuck it up and then say something completely false, like, "The version of TailwindCSS we're using is still in beta, let me try downgrading to the previous version."
I'm not saying it wasn't helpful. It probably cut 20% off the time it would have taken me to scaffold out the app myself, which is significant. But it certainly couldn't keep track of the context provided by the repo, even though it was creating that context itself.
Working with Copilot is like working with a very talented and fast junior developer whose methamphetamine addiction has been getting the better of it lately, and who has early onset dementia or a brain injury that destroyed their short-term memory.
-
AI tools are actually improving at a rate faster than most junior engineers I have worked with, and about 30% of junior engineers I have worked with never really "graduated" to a level that I would trust them to do anything independently, even after 5 years in the job. Those engineers "find their niche" doing something other than engineering with their engineering job titles, and that's great, but don't ever trust them to build you a bridge or whatever it is they seem to have been hired to do.
Now, as for AI, it's currently as good or "better" than about 40% of brand-new fresh from the BS program software engineers I have worked with. A year ago that number probably would have been 20%. So far it's improving relatively quickly. The question is: will it plateau, or will it improve exponentially?
Many things in tech seem to have an exponential improvement phase, followed by a plateau. CPU clock speed is a good example of that. Storage density/cost is one that doesn't seem to have hit a plateau yet. Software quality/power is much harder to gauge, but it definitely is still growing more powerful / capable even as it struggles with bloat and vulnerabilities.
The question I have is: will AI continue to write "human compatible" software, or is it going to start writing code that only AI understands, but people rely on anyway? After all, the code that humans write is incomprehensible to 90%+ of the humans that use it.
I’m seeing exactly the opposite. It used to be the junior engineers understood they had a lot to learn. However with AI they confidently try entirely wrong changes. They don’t understand how to tell when the ai goes down the wrong path, don’t know how to fix it, and it takes me longer to fix.
So far ai overall creates more mess faster.
Don’t get me wrong, it can be a useful tool you have to think of it like autocomplete or internet search. Just like those tools it provides results but the human needs judgement and needs to figure out how to apply the appropriate results.
My company wants metrics on how much time we’re saving with ai, but
- I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative
- it’s just another tool. There’s not really a defined task or set time. If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?
-
Writing code is the easiest part of my job. Why are you taking that away?
For some of us that’s more useful. I’m currently playing a DevSecOps role and one of the defining characteristics is I need to know all the tools. On Friday, I was writing some Java modules, then some groovy glue, then spent the after writing a Python utility. While im reasonably good about jumping among languages and tools, those context switches are expensive. I definitely want ai help with that.
That being said, ai is just a step up from search or autocomplete, it’s not magical. I’ve had the most luck with it generating unit tests since they tend to be simple and repetitive (also a major place for the juniors to screw up: ai doesn’t know whether the slop it’s pumping out is useful. You do need to guide it and understand it, and you really need to cull the dreck)
-
I would say that "replacing with AI assistance" is probably not what is actually happening. Is it economic factors reducing hiring. This isn't the first time it has happened and it won't be the last. The AI boosters are just claiming responsibility for marketing purposes.
It may also be self fulfilling. Our new ceo said all upcoming projects must save 15% using ai, and while we’re still hiring it’s only in India.
So 6 months from now we will have status reports talking about how we saved 15% in every project
-
Code reviews take up a lot of time, and if I know a lot of code in a review is AI generated I feel like I'm obliged to go through it with greater rigour, making it take up more time. LLM code is unaware of fundamental things such as quirks due to tech debt and existing conventions. It's not great.
Code reviews seem like a good opportunity for an LLM. It seems like they would be good at it. I’ve actually spent the last half hour googling for tools.
I’ve spent literally a month in reviews for this junior guy on one stupid feature, and so much of it has been so basic. It’s a combination of him committing ai slop without understanding or vetting it, and being too junior to consider maintainability or usability. It would have saved so much of my time if ai could have done some of those review cycles without me
-
Just the other day I wasted 3 min trying to get AI to sort 8 lines alphabetically.
I had to sort over 100 lines of data hardcoded into source (don’t ask) and it was a quick function in my IDE.
I feel like “sort” is common enough everywhere that AI should quickly identify the right Google results, and it shouldn’t take 3 min
-
That's still not actually knowing anything. It's just temporarily adding more context to its model.
And it's always very temporary. I have a yarn project I'm working on right now, and I used Copilot in VS Code in agent mode to scaffold it as an experiment. One of the refinements I included in the prompt file to build it is reminders throughout for things it wouldn't need reminding of if it actually "knew" the repo.
- I had to constantly remind it that it's a yarn project, otherwise it would inevitably start trying to use NPM as it progressed through the prompt.
- For some reason, when it's in agent mode and it makes a mistake, it wants to delete files it has fucked up, which always requires human intervention, so I peppered the prompt with reminders not to do that, but to blank the file out and start over in it.
- The frontend of the project uses TailwindCSS. It could not remember not to keep trying to downgrade its configuration to an earlier version instead of using the current one, so I wrote the entire configuration for it by hand and inserted it into the prompt file. If I let it try to build the configuration itself, it would inevitably fuck it up and then say something completely false, like, "The version of TailwindCSS we're using is still in beta, let me try downgrading to the previous version."
I'm not saying it wasn't helpful. It probably cut 20% off the time it would have taken me to scaffold out the app myself, which is significant. But it certainly couldn't keep track of the context provided by the repo, even though it was creating that context itself.
Working with Copilot is like working with a very talented and fast junior developer whose methamphetamine addiction has been getting the better of it lately, and who has early onset dementia or a brain injury that destroyed their short-term memory.
Adding context is “knowing more” for a computer program.
Maybe it’s different in VS code vs regular VS, because I never get issues like what you’re describing in VS. Haven’t really used it in VS Code.
-
Code reviews seem like a good opportunity for an LLM. It seems like they would be good at it. I’ve actually spent the last half hour googling for tools.
I’ve spent literally a month in reviews for this junior guy on one stupid feature, and so much of it has been so basic. It’s a combination of him committing ai slop without understanding or vetting it, and being too junior to consider maintainability or usability. It would have saved so much of my time if ai could have done some of those review cycles without me
This has been solved for over a decade. Include a linter and static analysis stage in the build pipeline. No code review until the checkbox goes green (or the developer has a specific argument for why a particular finding is a false positive)
-
I’m seeing exactly the opposite. It used to be the junior engineers understood they had a lot to learn. However with AI they confidently try entirely wrong changes. They don’t understand how to tell when the ai goes down the wrong path, don’t know how to fix it, and it takes me longer to fix.
So far ai overall creates more mess faster.
Don’t get me wrong, it can be a useful tool you have to think of it like autocomplete or internet search. Just like those tools it provides results but the human needs judgement and needs to figure out how to apply the appropriate results.
My company wants metrics on how much time we’re saving with ai, but
- I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative
- it’s just another tool. There’s not really a defined task or set time. If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?
I've always had problems with junior engineers (self included) going down bad paths, since before there was Google search - let alone AI.
So far ai overall creates more mess faster.
Maybe it is moving faster, maybe they do bother the senior engineers less often than they used to, but for throw-away proof of concept and similar stuff, the juniors+AI are getting better than the juniors without senior support used to be... Is that a good direction? No. When the seniors are over-tasked with "Priority 1" deadlines (nothing new) does this mean the juniors can get a little further on their own and some of them learn from their own mistakes? I think so.
Where I started, it was actually the case that the PhD senior engineers needed help from me fresh out of school - maybe that was a rare circumstance, but the shop was trying to use cutting edge stuff that I knew more about than the seniors. Basically, everything in 1991 was cutting edge and it made the difference between getting something that worked or having nothing if you didn't use it. My mentor was expert in another field, so we were complimentary that way.
My company (now) wants metrics on a lot of things, but they also understand how meaningless those metrics can be.
I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative
Shame. There was a time that people dug out of their own messes, I think you learn more, faster that way. Still, I agree - since 2005 I have spend a lot of time taking piles of Matlab, Fortran, Python that have been developed over years to reach critical mass - add anything else to them and they'll go BOOM - and translating those into commercially salable / maintainable / extensible Qt/C++ apps, and I don't think I ever had one "mentee" through that process who was learning how to follow in my footsteps, the organizations were always just interested in having one thing they could sell, not really a team that could build more like it in the future.
it’s just another tool.
Yep.
If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?
Speaking of meaningless metrics, how many people ask you for Lines Of Code counts, even today?___
-
Adding context is “knowing more” for a computer program.
Maybe it’s different in VS code vs regular VS, because I never get issues like what you’re describing in VS. Haven’t really used it in VS Code.
Are you using agent mode?
