Well, for the relatively small sample of Kubernetes experts I interviewed, basically any topic beyond "you use this tool" was a disaster, including Kubernetes knowledge.
I am not selective, it's not like I expect a specific skillset, but what would you think if someone with a decade of platform security doesn't understand cryptography and supply chain, Linux permissions, Kubernetes foundational concepts, container isolation or networking? At some point the question is legitimate, what are you expert in? The answer I have been able to give myself so far is "stitching together services that do stuff" and "recommend what the documentation/standard recommends".
I consider myself satisfied to have somewhat decent knowledge in some of those areas, I am not expecting someone understanding all of that, but none of them? Maybe from someone who just joined the industry.

Does a senior mechanic need to understand the physics of piston design to be a great mechanic

I would argue that if senior mechanic doesn't understand the physics of piston design at least on some degree he's not a great mechanic. Obviously mechanic doesn't need understanding on metallurgy, CAD models and a ton of other deeper level stuff just like an IT engineer doesn't need to know on a deep level how circuit boards are designed or how CPU die manufacturing process works. But both benefit greatly when they understand why something is built the way it is.

I'm also an systems engineer of sorts and have worked with software engineers. And I've had requests like "Can't you just set 'bind-address = 0.0.0.0 on mysql-server and disable firewall" on a directly internet-facing machine and then received complaints when I'm "making things more difficult" from "senior software" -titles. Sure, I can't write the code they're doing, or at least it would take me a crapload of more time to do that but on the other hand there's guys who have so very narrow understanding on anything they work with that it makes me wonder how they can do their work at all in the first place.

Of course no one can master everything in any field but I find it concerning that a lot of guys just press the buttons more or less randomly until their thing works without any clue on what they actually did and how it might affect on different parts of the house of cards they're building.

The main factor, IMO, is that everyone wants good engineers but good engineers don't change jobs that often.

Meaning most of the candidates you interview will suck in one way or another.

And everyone calls themselves "senior" nowadays.

I have the opposite experience, when I was doing interviews I just skipped the very obviously underskilled people (which, IIRC were in the single digits) and interviewed pretty much everyone.

For context, I'm the main architect and dev of the company I was hiring for. Most of the candidates were horrible.

I get what you’re saying, but also see the other side - these services exist and aren’t ever going away, so the level of knowledge you need about these to use them at least competently is significantly reduced.

What their existence does mean is that there are thousands of developers who wouldn’t ever touch or learn any of this stuff previously are now actually learning it and using it. That’s a positive thing. Not everyone needs to be an expert on the inner workings of everything that a service provides unless you’re specifically looking for an expert.

Also…..people lie on CVs and cover letters. If your ad has buzzwords and technology X, Y, and Z, then you should expect people with little to no knowledge of at least one of those things to have all 3 on their resume.

I will say more, the same exact team that spends time managing EKS clusters could manage self-managed clusters and have money to spare for additional hires.

Your suggestions is a large expansion of skillset needed for your alternative to the cloud solution. Your own experience in attempting to hire workers should point to the reason thats a bad idea. You're going to need even higher skilled people, and they are going to ask for significantly more money.

I wouldn't say it's a large expansion of skillset, meaning it's not massive. But yes, indeed it is problematic to find people. It is because this is a vicious circle in which companies are digging their own graves by eliminating a market for those people, which in turn means that those who would want to hire some can't find them easily, leading to outsourcing instead. Do this for 15 years across the whole industry and it stops being an option, which is pretty much where we are today.
That said, training and upskilling is always a possibility for companies who invest on their own employees and are playing the long game...

I partially agree, but not only we are looking for experts of that thing, we are also looking for security experts, and security knowledge is very much meta-knowledge.
A software developer might not care at all about - say - how the CI/CD works, because all they care is that the thing builds the code. A security expert generally has a broader scope, and their job is not functional, which means their job is exactly understanding the thing to be able to model the risks around it. So they might not care of all the tools used in that CI/CD or the exact details of the steps, but they should understand the execution flow, the way third party dependencies are pulled, verified, consumed, the authorization model etc.

There is no such thing of security professional who doesn't understand - at least from an academic point of view - the overall setup of a thing they worked with.

If I take the image attestation example I made in the post, I consider the "inner workings" to be the cryptographic details, such as ciphers and their working mechanisms, or the exact details of the way that attestation can be verified offline, or what exactly is computed and how. I am OK with someone not knowing this. But not understanding the whole flow? Well, without this what's left? Copying the 3 lines of code that do something from the Github documentation? Any software engineer can very much do that, what is your contribution as a security specialist?

…..people lie on CVs and cover letters. If your ad has buzzwords and technology X, Y, and Z

Totally agree. It is very likely, although the more people I interview, the more I think that they are not lying from their perspective. It's that people can legitimately make a career today by stitching together stuff with scotch tape, spending years by just by doing that and effectively have little to show for those years. But from their perspective, they might be experienced in that stuff, maybe?

Everyone calls themselves senior because that's the only type of position recruiters look for.

I'm a mid level dev, but I'm encouraged by recruiters to apply for senior positions because their clients are actually looking for a range of levels

Yeah, that's true, everyone thinks they want a senior where usually someone who's not a straight up junior is more than enough. And a fast learning and motivated junior is the best you can get, IMO, though those are pretty rare as well.

A lot of this has to do with recruiters. I've been interviewing for a few years at my company such with as many different sets of recruiters, from recruiting firms to our corporate recruiters, to ones we hired ourselves. Our corporate recruiters handed over garbage candidates who we could often tell wouldn't work out after the first 10 min of the interview, whereas the other two groups of recruiters would do a good job filtering so we'd get than a 50% hit rate on our first round. Unfortunately, we promoted our recruiters once the need for talent dropped (or they moved on to a recruiter firm), and now they're unwilling to go back to recruiting.

The quality of your recruiter matters quite a bit, so you'll want to find someone who is experienced hiring a certain type of person so they know what to look for.

Exactly. We don't hire "junior" positions, because all the midlevels are juniors, all seniors are mid-level, and seniors don't apply. I'm a senior and a recruiter found me, I didn't apply (at least not to this company).

Ours is really simple, like something any somewhat competent engineer could complete in half the time we provide after going through the tutorial on the framework's website. Yet so many people fail, even when they claim to have years of experience with the framework.

There are a ton of terrible applicants out there.

I disagree. On paper that sounds good, but I firmly believe good engineers are curious, so they'll learn a lot more than necessary to do the job.

For example, when I worked at a company that designed antennas as a software engineer (built something tangentially related), I didn't need to know anything about electrical engineering, but I was curious so I asked a ton of questions and now I know a fair amount about EE. These days I work in a very different domain and still ask a ton of questions to our domain experts. In my own field, I look into all kinds of random things tangentially related to the tools I use. In each case, that curiosity has come in handy at some point or another.

In each role, I can tell who's there to clock in and clock out vs who is genuinely curious and looking to improve, and it's the latter group who tend to produce the best work and go on to great roles after leaving our company, while the 9-5 warriors who just focus on the requirements tend to do pretty mediocre when it comes to advancement.

When I hire, I look for that curiosity because you never know what you'll need to know to fix a prod issue quickly. My esoteric knowledge about SSH helped keep my team productive for a few days when IT was being slow revolving our issue, and likewise we've had quick resolution to prod bugs because someone on the team knew something random that ended up being relevant. That's what I mean when I say I look for a diverse team, I want people with different strengths who all actively seek to improve so we'll have a good shot at handling whatever comes down the pipe (and we get a lot of random stuff, from urgently needing to embed 3D modeling tools into our reporting app to needing to embed complex C++ simulation code or rewrite Fortran code into our largely CRUD Python app).

Most of these cases of "focus on one niche" are often symptoms of lacking curiosity and just wanting to tick boxes to quality for a role. I'd much rather someone miss a few important boxes but tick a lot of random ones because they're curious; they'll take longer to on-board, but they'll likely be more useful long term.

I don't work in the security space, but I think the same applies to most technical fields. Breadth of knowledge in an individual provides depth of knowledge in a team.

I somewhat disagree here, but also somewhat agree.

In my org, we get a lot of requirements that require very different skillsets. For the first 2-3 years, our task list was mostly CRUD stuff with some domain specific logic, but otherwise a boring web app. In the last 1-2 years, we have:

• ported a Fortran simulation to Python
• embedded a C++ simulation in Python
• created a 3D UX for our previously 2D only app (lots of 3D logic on both FE and BE)
• implemented a machine learning algorithm to train our simulations

If I hired only for the work I'd seen in the past, we'd be completely unfit to handle this workload since we'd mostly have people who are really good at building CRUD apps (so DB optimization and quick UX building).

On the flipside, we cut off huge swaths of work so people don't need to wear too many hats. We have:

• dedicated devOPs - handles everything from trst pipelines to prod deployments
• dedicated QA - manual and automated app-level testing - devs still do unit testing
• dedicated product teams who handle feature requirements and documentation
• dedicated UX team to produce designs for FE engineers to implement

So our devs only need to worry about development, but they also need a broad skillset in that domain, from everything from local tooling to working in different domains. We hire a diverse set of candidates, some with a heavy math background, some with design experience, and some with low level programming experience, because we never know what projects we'll get or who will suddenly leave the org.

Instead of hiring gurus, I think you want a diverse set of curious "regular" people. Maybe one person is really good with working in different number bases (and 0xD6 in decimal is something they know off the top of their head), another is really good w/ databases, etc. None of those would know everything, but they're all curious and picked up random stuff from their career because they asked a lot of questions.

Hiring the right guru is hard, having the equivalent of a guru across a diverse time is a lot more tractible, and maybe one will become that guru you need after cross pollinating with the team.

I 100% agree.

The best mechanics can track down an issue by reasoning about what could be causing it, and understanding how pistons work can help deduce whether that knocking is actually the engine or something else entirely. They probably didn't learn that from their official training, but instead worked with some guy who used to work at a car manufacturer or something and picked their brain.

The best engineers are curious and jump on opportunities to learn more.

Yeah I don't think we actually disagree much here.

I think my angle is just slightly different? I see that ease of access (eg cloud) make it possible for a lot more uncurious and clock-out people to enter the field and pass as competent. To be honest, even the modest introduction of auto-formatting editors are easy to see as good and useful, but I also feel that they allowed shoddy work to look passable at first glance. AI will make this a lot worse.

But as for the actual people who have it in them to be competent, people that were always there and still are, cloud is not going to make them worse.

The best mechanics can track down an issue by reasoning about what could be causing it

Same principle works with IT. I do and have done sysadmin stuff for quite a while and there's always some random software or whatever I've never heard of and someone comes and asks me to fix it. Then you start to ask questions, "what exactly doesn't work", "can you show me what you're doing", "what should happen when you press that button", "can you show settings on that thing" and so on. Then you can start to dig down, does the server they're using respond to ping, does DNS resolve (it's always DNS after all), does that thing work on the next workstation, when did the problem appear and was there some other maintenance or changes going on at that time and so on.

Same principle, just start to reason the whole thing from bottom up, check everything you come across untill you find something which doesn't work and then do what's needed to fix that, rinse and repeat until the problem goes away and make sure that what you're doing won't cause new problems. Just the tools are different, the mindset is more or less the same.