You shouldn't think of "AI" as intelligent and ask it to do something tricky. The boring stuff that's mostly just typing, that's what you get the LLMs to do. "Make a DTO for this table <paste>" "Interface for this JSON <paste>"

I just have a bunch of conversations going where I can paste stuff into and it will generate basic code. Then it's just connecting things up, but that's the fun part anyway.

That's still not actually knowing anything. It's just temporarily adding more context to its model.

And it's always very temporary. I have a yarn project I'm working on right now, and I used Copilot in VS Code in agent mode to scaffold it as an experiment. One of the refinements I included in the prompt file to build it is reminders throughout for things it wouldn't need reminding of if it actually "knew" the repo.

• I had to constantly remind it that it's a yarn project, otherwise it would inevitably start trying to use NPM as it progressed through the prompt.
• For some reason, when it's in agent mode and it makes a mistake, it wants to delete files it has fucked up, which always requires human intervention, so I peppered the prompt with reminders not to do that, but to blank the file out and start over in it.
• The frontend of the project uses TailwindCSS. It could not remember not to keep trying to downgrade its configuration to an earlier version instead of using the current one, so I wrote the entire configuration for it by hand and inserted it into the prompt file. If I let it try to build the configuration itself, it would inevitably fuck it up and then say something completely false, like, "The version of TailwindCSS we're using is still in beta, let me try downgrading to the previous version."

I'm not saying it wasn't helpful. It probably cut 20% off the time it would have taken me to scaffold out the app myself, which is significant. But it certainly couldn't keep track of the context provided by the repo, even though it was creating that context itself.

Working with Copilot is like working with a very talented and fast junior developer whose methamphetamine addiction has been getting the better of it lately, and who has early onset dementia or a brain injury that destroyed their short-term memory.

I’m seeing exactly the opposite. It used to be the junior engineers understood they had a lot to learn. However with AI they confidently try entirely wrong changes. They don’t understand how to tell when the ai goes down the wrong path, don’t know how to fix it, and it takes me longer to fix.

So far ai overall creates more mess faster.

Don’t get me wrong, it can be a useful tool you have to think of it like autocomplete or internet search. Just like those tools it provides results but the human needs judgement and needs to figure out how to apply the appropriate results.

My company wants metrics on how much time we’re saving with ai, but

• I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative
• it’s just another tool. There’s not really a defined task or set time. If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?

For some of us that’s more useful. I’m currently playing a DevSecOps role and one of the defining characteristics is I need to know all the tools. On Friday, I was writing some Java modules, then some groovy glue, then spent the after writing a Python utility. While im reasonably good about jumping among languages and tools, those context switches are expensive. I definitely want ai help with that.

That being said, ai is just a step up from search or autocomplete, it’s not magical. I’ve had the most luck with it generating unit tests since they tend to be simple and repetitive (also a major place for the juniors to screw up: ai doesn’t know whether the slop it’s pumping out is useful. You do need to guide it and understand it, and you really need to cull the dreck)

It may also be self fulfilling. Our new ceo said all upcoming projects must save 15% using ai, and while we’re still hiring it’s only in India.

So 6 months from now we will have status reports talking about how we saved 15% in every project

Code reviews seem like a good opportunity for an LLM. It seems like they would be good at it. I’ve actually spent the last half hour googling for tools.

I’ve spent literally a month in reviews for this junior guy on one stupid feature, and so much of it has been so basic. It’s a combination of him committing ai slop without understanding or vetting it, and being too junior to consider maintainability or usability. It would have saved so much of my time if ai could have done some of those review cycles without me

I had to sort over 100 lines of data hardcoded into source (don’t ask) and it was a quick function in my IDE.

I feel like “sort” is common enough everywhere that AI should quickly identify the right Google results, and it shouldn’t take 3 min

Adding context is “knowing more” for a computer program.

Maybe it’s different in VS code vs regular VS, because I never get issues like what you’re describing in VS. Haven’t really used it in VS Code.

This has been solved for over a decade. Include a linter and static analysis stage in the build pipeline. No code review until the checkbox goes green (or the developer has a specific argument for why a particular finding is a false positive)

I've always had problems with junior engineers (self included) going down bad paths, since before there was Google search - let alone AI.

So far ai overall creates more mess faster.

Maybe it is moving faster, maybe they do bother the senior engineers less often than they used to, but for throw-away proof of concept and similar stuff, the juniors+AI are getting better than the juniors without senior support used to be... Is that a good direction? No. When the seniors are over-tasked with "Priority 1" deadlines (nothing new) does this mean the juniors can get a little further on their own and some of them learn from their own mistakes? I think so.

Where I started, it was actually the case that the PhD senior engineers needed help from me fresh out of school - maybe that was a rare circumstance, but the shop was trying to use cutting edge stuff that I knew more about than the seniors. Basically, everything in 1991 was cutting edge and it made the difference between getting something that worked or having nothing if you didn't use it. My mentor was expert in another field, so we were complimentary that way.

My company (now) wants metrics on a lot of things, but they also understand how meaningless those metrics can be.

I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative

Just a moment...

(clip.cafe)

Shame. There was a time that people dug out of their own messes, I think you learn more, faster that way. Still, I agree - since 2005 I have spend a lot of time taking piles of Matlab, Fortran, Python that have been developed over years to reach critical mass - add anything else to them and they'll go BOOM - and translating those into commercially salable / maintainable / extensible Qt/C++ apps, and I don't think I ever had one "mentee" through that process who was learning how to follow in my footsteps, the organizations were always just interested in having one thing they could sell, not really a team that could build more like it in the future.

it’s just another tool.

Yep.

If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?

Speaking of meaningless metrics, how many people ask you for Lines Of Code counts, even today?___

Are you using agent mode?

Most ides do the boring stuff with templates and code generation for like a decade so that's not so helpful to me either but if it works for you.

That's kind of outside the software development discussion but glad you're enjoying it.

Shame. There was a time that people dug out of their own messes, I think you learn more, faster

Yes, that’s how we became senior guys. But when you have deadlines that you’re both on the hook for and they’re just floundering, you can only give them so much opportunity. I’ve had too many arguments with management about letting them merge and I’m not letting that ruin my code base

Speaking of meaningless metrics, how many people ask you for Lines Of Code counts, even today?

We have a new VP collecting metrics on everyone, including lines of code, number of merge requests, times per day using ai, days per week in the office vs at home

Not really.

Linter in the build pipeline is generally not useful because most people won’t give results time or priority. You usually can’t fail the build for lint issues so all it does is fill logs. I usually configure a linter and prettifier in a precommit hook, to shift that left. People are more willing to fix their code in small pieces as they try to commit.

But this is also why SonarQube is a key tool. The scanners are lint-like, and you can even import some lint output. But the important part is it tries to prioritize them, score them, and enforce a quality gate based on them. I usually can’t fail a build for lint errors but SonarQube can if there are too many or too priority, or if they are security related.

But this is not the same as a code review. If an ai can use the code base as context, it should be able to add checks for consistency and maintainability similar to the rest of the code. For example I had a junior developer blindly follow the AI to use a different mocking framework than the rest of the code, for no reason other than it may have been more common in the training data. A code review ai should be able to notice that. Maybe this is too advanced for current ai, but the same guy blindly followed ai to add classes that already existed. They were just different enough that SonarQube didn’t flag is as duplicate code but ai ought to be able to summarize functionality and realize they were the same. Or I wonder if ai could do code organization? Junior guys spew classes and methods everywhere without any effort in organizing like with like, so someone can maintain it all. Or how about style? I hope yo never revisit style wars but when you’re modifying code you really need to follow style and naming of what’s already there. Maybe ai code review can pick up on that

As a developer

• I can jot down a bunch of notes and have ai turn it into a reasonable presentation or documentation or proposal
• zoom has an ai agent which is pretty good about summarizing a meeting. It usually just needs minor corrections and you can send it out much faster than taking notes
• for coding I mostly use ai like autocomplete. Sometimes it’s able to autocomplete entire code blocks
• for something new I might have ai generate a class or something, and use it as a first draft where I then make it work