Skip to content

Using Clouds for too long might have made you incompetent

Technology
47 20 0
  • That is technically correct in a way, but I'll argue very wrong in a meaningful way.

    Cloud services are meant to let you focus less on the plumbing, so naturally many skills in that will not be developed, and skills adjacent to it will be less developed.

    Buttttt you must assume effort remains constant!

    So you get to focus more on other things now. E.g. functional programming, product thinking, rapid prototyping, API stuff, breadth of languages, etc. I bet the seniors you are missing X and Y in have bigger Zs and also some Qs that you may not be used to consider, or have the experience to spot and evaluate.

    Mind you that my take and experience is specifically in the context of security.

    I struggle to make the parallel that you suggest (which might work for some areas) with a security engineer.

    Say, a person learned to brainlessly parrot that pods need to have setting x or z. If they don't understand them, they can't offer meaningful insight in cases where that's not possibile (which might be specific), they can't provide a solid risk analysis etc.

    What is the counterpart to this gap?
    Because I struggle to see it. Breadth of areas where this superficial knowledge is available is useless, IMHO.

  • My take on how a decade (or more) of using cloud services for everything has seemingly deskilled the workforce.

    Just recently I found myself interviewing senior security engineers just to realize that in many cases they had absolutely no idea about how the stuff they supposedly worked with, actually worked.

    This all made me wonder, is it possible that over-reliance on cloud services for everything has massively deskilled the engineering workforce? And if it is so, who is going to be the European clouds, so necessary for EU's digital sovereignty?

    I did not copy-paste the post in here because of the different writing style, but I get no benefit whatsoever from website visits.

    I think its actually that most people generally don't really understand most things beyond the minimal level necessary to get by. Now that the tech industry isn't just a bunch of nerds you're increasingly more likely to encounter people who are temperamentally disinclined to seek understanding of those details.

  • That's the thing! I think it wouldn't be conceivable that your "principal engineer" (real position for one of the people) doesn't understand the basic theory of the stuff they are implementing. Now it feels you can instead work years and years just shuffling configuration and pressing buttons, leading to "senior" people who didn't gather actual years of experience.

    I don't want to pretend I am outside this logic. I am very much part of this problem myself, having started my career 10 years ago. I do despise cloud services though (if anything, they are super boring), so I tend to work with other stuff. But I could 100% just click buttons and parrot standard and keep accruing empty years of experience...

    I agree with your lack of affection for cloud services, but I think your view might be a little skewed here. Does a senior mechanic need to understand the physics of piston design to be a great mechanic, or just gather years of experience fixing problems with the whole system that makes up the car?

    I'm a Senior Systems engineer. I know very little about kernel programming or OS design, but i know how the packages and applications work together and where problems might arise in how they interact. Software Engineers might not know how or don't want to spend time to set up the infrastructure to host their applications, so they rely on me to do it for them, or outsource my job to someone else's computer.

  • This is quite a trite argument from my point of view. Also, this is from the perspective of the business, which I don't particularly care about, and I tend to look from the perspective of the worker.

    Additionally, the cloud allows to scale quickly, but the fact that it allows to delegate everything is a myth. It's so much a myth that you see companies running fully on cloud with an army on people in platform teams and additionally you get finops teams, entire teams whose job is optimizing the spend of cloud.
    Sure, when you start out it's 100% reasonable to use cloud services, but in the medium-long term, it's an incredibly poor investment, because you still need people to administer the cloud plus, you need to pay a huge premium for the services you buy, which your workforce now can't manage or build anymore. This means you still pay people to do work which is not your core business, but now they babysit cloud services instead of the actual infra, and you are paying twice.

    Cloud exploded during the times of easy money at no interest, where startups had to build some stuff, IPO and then explode without ever turning a single dollar of profit. It's a model that fits perfect in that context.

    I get you that it's easy to over-provision in the cloud, but you can't return an on-prem server. A cloud VM, just shut it down and you're done.

    AWS talks about minimizing undifferentiated heavy lifting as a reason to adopt managed services and I find that largely to be true. The majority of companies aren't differentiating their services via some low-level technology advantage that allows them to cost less. It's a different purchasing model, a smoother workflow, or a unique insight into data. The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.

  • This is quite a trite argument from my point of view. Also, this is from the perspective of the business, which I don't particularly care about, and I tend to look from the perspective of the worker.

    Additionally, the cloud allows to scale quickly, but the fact that it allows to delegate everything is a myth. It's so much a myth that you see companies running fully on cloud with an army on people in platform teams and additionally you get finops teams, entire teams whose job is optimizing the spend of cloud.
    Sure, when you start out it's 100% reasonable to use cloud services, but in the medium-long term, it's an incredibly poor investment, because you still need people to administer the cloud plus, you need to pay a huge premium for the services you buy, which your workforce now can't manage or build anymore. This means you still pay people to do work which is not your core business, but now they babysit cloud services instead of the actual infra, and you are paying twice.

    Cloud exploded during the times of easy money at no interest, where startups had to build some stuff, IPO and then explode without ever turning a single dollar of profit. It's a model that fits perfect in that context.

    At least where I work, our cloud team is ~35 people who manage the whole thing.

    The datacenter team? In the hundreds.

    Cloud is not the answer to every infra problem, but the flexibility, time to market, and lifecycle burden are easily beneficial weighed against finops. I’m an Azure engineer myself, it’s no comparison the benefits to a managed solution vs rolling your own DC for a lot of regular business workloads and solutions. Beyond that personally I’ve been able to skill up in areas I wouldn’t be able to otherwise if I was stuck troubleshooting bad cables, rebuilding a dead RAID array, or planning VMWare scaling nonsense.

  • Mind you that my take and experience is specifically in the context of security.

    I struggle to make the parallel that you suggest (which might work for some areas) with a security engineer.

    Say, a person learned to brainlessly parrot that pods need to have setting x or z. If they don't understand them, they can't offer meaningful insight in cases where that's not possibile (which might be specific), they can't provide a solid risk analysis etc.

    What is the counterpart to this gap?
    Because I struggle to see it. Breadth of areas where this superficial knowledge is available is useless, IMHO.

    Because a security engineer focused on cloud would rightfully say "pod security is not my issue, I'm focused on protecting the rest of our world from each pod itself.". With AWS as example:
    If they then analyze the IAM role structures and to deep into where the pod runs (e.g. shared ec2 vs eks) etc. then it would just be a matter of different focus.

    Cloud security is focused on the infrastructure - looks like you're looking for a security engineer focused on the dev side.

    If they bring neither to the table then I'm with you - but I don't see how "the cloud" is at fault here... especially for security the world as full of "following the script" people long before cloud was a thing.

  • My take on how a decade (or more) of using cloud services for everything has seemingly deskilled the workforce.

    Just recently I found myself interviewing senior security engineers just to realize that in many cases they had absolutely no idea about how the stuff they supposedly worked with, actually worked.

    This all made me wonder, is it possible that over-reliance on cloud services for everything has massively deskilled the engineering workforce? And if it is so, who is going to be the European clouds, so necessary for EU's digital sovereignty?

    I did not copy-paste the post in here because of the different writing style, but I get no benefit whatsoever from website visits.

    I went through hiring several times at several companies, being on the interviewer side.

    Typically it's not the talent pool as much as what the company has to offer and how much they're willing to pay. I referred top notch engineer friends, and they never made it past HR. A couple were rejected without interview because they asked too high of a salary, despite asking under market average. The rest didn't pass HR on personnality or not having all the "requirements", because the really good engineers are socially awkward and demand flexibility and are honest on the résumé/CV, or are self taught and barely have high-school graduation on there (just like me).

    I've literally seen the case of: they want to hire another me, but ended up in a situation where: I wouldn't apply for the position myself, and even if I did, I wouldn't make it to the interview stage where I'd talk to myself and hire myself.

    Naturally the candidates that did make it to me weren't great. Those are the people that do the bare minimum, have studied every test question (without understanding), vibe code everything, typically on the younger and very junior side. They're very good at passing HR, and very bad at their actual job.

    It's not the technology, it's the companies that hire that ultimately steers the market and what people study for. Job requirements are ridiculous, HR hires engineers on personnality like they're shopping for yet another sales associate, now it takes 6 rounds of interviews for an entry level position at a startup. VC startups continue to pay wildly inflated wages to snatch all the top talent while established companies are laying off as much IT staff as possible to maximize profits.

  • My take on how a decade (or more) of using cloud services for everything has seemingly deskilled the workforce.

    Just recently I found myself interviewing senior security engineers just to realize that in many cases they had absolutely no idea about how the stuff they supposedly worked with, actually worked.

    This all made me wonder, is it possible that over-reliance on cloud services for everything has massively deskilled the engineering workforce? And if it is so, who is going to be the European clouds, so necessary for EU's digital sovereignty?

    I did not copy-paste the post in here because of the different writing style, but I get no benefit whatsoever from website visits.

    I'm not in any way, shape, or form an engineer so I don't really understand the exact details of your post.

    However, you post reminded me of a really good episode of a podcast called Hidden Brain. In it the host, discusses the topic of knowledge with a cognitive scientist.

    At one point, they talk about how sophisticated technology has gotten that people don't know how to solve problems if that technology brakes, especially since technology is getting so good that it makes fewer mistakes. They use an airplane as an example in which an experienced pilot forgot how to get out of a nosedive and crashed the plane. On a smaller scale, the host mentioned that he has a hard time navigating if his phone's GPS doesn't work.

    Its a really interesting listen if you have the chance.

  • I'm reminded of when my boss asked me whether our entry test was too hard after getting several submissions that wouldn't even run.

    Sometimes prospective employees are just shit.

    I got asked the same. I simply pointed out the test is a reproduction of last week's bug that took down prod at 2am and got paged to fix, and is therefore as realistic as it gets of what they'll need to be able to handle.

    It's always DNS, everyone should know that.

  • I got asked the same. I simply pointed out the test is a reproduction of last week's bug that took down prod at 2am and got paged to fix, and is therefore as realistic as it gets of what they'll need to be able to handle.

    It's always DNS, everyone should know that.

    It’s always DNS, everyone should know that.

    It's not DNS. There's no way it is DNS. It's not technically possible for it to be DNS.

    And it's always DNS.

  • At least where I work, our cloud team is ~35 people who manage the whole thing.

    The datacenter team? In the hundreds.

    Cloud is not the answer to every infra problem, but the flexibility, time to market, and lifecycle burden are easily beneficial weighed against finops. I’m an Azure engineer myself, it’s no comparison the benefits to a managed solution vs rolling your own DC for a lot of regular business workloads and solutions. Beyond that personally I’ve been able to skill up in areas I wouldn’t be able to otherwise if I was stuck troubleshooting bad cables, rebuilding a dead RAID array, or planning VMWare scaling nonsense.

    But those are absolutely not the only 2 levels. Server rental can be managed easily by the same infra team who manages the cloud, for a fraction of cost.

    I will say more, the same exact team that spends time managing EKS clusters could manage self-managed clusters and have money to spare for additional hires.

  • I went through hiring several times at several companies, being on the interviewer side.

    Typically it's not the talent pool as much as what the company has to offer and how much they're willing to pay. I referred top notch engineer friends, and they never made it past HR. A couple were rejected without interview because they asked too high of a salary, despite asking under market average. The rest didn't pass HR on personnality or not having all the "requirements", because the really good engineers are socially awkward and demand flexibility and are honest on the résumé/CV, or are self taught and barely have high-school graduation on there (just like me).

    I've literally seen the case of: they want to hire another me, but ended up in a situation where: I wouldn't apply for the position myself, and even if I did, I wouldn't make it to the interview stage where I'd talk to myself and hire myself.

    Naturally the candidates that did make it to me weren't great. Those are the people that do the bare minimum, have studied every test question (without understanding), vibe code everything, typically on the younger and very junior side. They're very good at passing HR, and very bad at their actual job.

    It's not the technology, it's the companies that hire that ultimately steers the market and what people study for. Job requirements are ridiculous, HR hires engineers on personnality like they're shopping for yet another sales associate, now it takes 6 rounds of interviews for an entry level position at a startup. VC startups continue to pay wildly inflated wages to snatch all the top talent while established companies are laying off as much IT staff as possible to maximize profits.

    I totally agree with you, but I don't think this is the specific case.
    Most of the rejections in our case (which I can see) on the preliminary screening were based on lacking CV skills. Which is stupid in its own way, but at least makes sense assuming we are looking for those skills specifically.

    For the rest, the company is a remote company paying good salaries for the European market, I would say slightly above market average in many metrics.

    I will sift more into the rejections, but from what I have seen, almost all those who had the screening phone call made it to the interview (I.e., rejections were mostly cv-based).

  • I'm not in any way, shape, or form an engineer so I don't really understand the exact details of your post.

    However, you post reminded me of a really good episode of a podcast called Hidden Brain. In it the host, discusses the topic of knowledge with a cognitive scientist.

    At one point, they talk about how sophisticated technology has gotten that people don't know how to solve problems if that technology brakes, especially since technology is getting so good that it makes fewer mistakes. They use an airplane as an example in which an experienced pilot forgot how to get out of a nosedive and crashed the plane. On a smaller scale, the host mentioned that he has a hard time navigating if his phone's GPS doesn't work.

    Its a really interesting listen if you have the chance.

    Thanks, indeed I think there are many parallels with other areas. I will check it out.

  • Because a security engineer focused on cloud would rightfully say "pod security is not my issue, I'm focused on protecting the rest of our world from each pod itself.". With AWS as example:
    If they then analyze the IAM role structures and to deep into where the pod runs (e.g. shared ec2 vs eks) etc. then it would just be a matter of different focus.

    Cloud security is focused on the infrastructure - looks like you're looking for a security engineer focused on the dev side.

    If they bring neither to the table then I'm with you - but I don't see how "the cloud" is at fault here... especially for security the world as full of "following the script" people long before cloud was a thing.

    I mean, the person in question had "hardening EKS" on their CV. EKS still means that the whole data plane is your responsibility. How can you harden a cluster without understanding the foundation of container security (isolation primitives, capabilities, etc.)? Workload security is very much part of the job.

    I mean the moment some pod will need to run with some privilege (say, a log forwarder which gets host logs), and you need to "harden" the cluster, what do you do if you don't understand the concept of capabilities? I will tell you what, because I asked this very question, and the answer was "copy the logs elsewhere", which is the "make it work with the hammer solution" that again shows the damage of not understanding.

    I am with you about different scopes, skillsets etc. But here we were interviewing people with a completely matching skillset on paper.

  • I get you that it's easy to over-provision in the cloud, but you can't return an on-prem server. A cloud VM, just shut it down and you're done.

    AWS talks about minimizing undifferentiated heavy lifting as a reason to adopt managed services and I find that largely to be true. The majority of companies aren't differentiating their services via some low-level technology advantage that allows them to cost less. It's a different purchasing model, a smoother workflow, or a unique insight into data. The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.

    A cloud VM, just shut it down and you're done.

    If this flexibility is needed, and it's an "if", a dedicated server does the same. But even a cloudVM is already lower level compared to other services (which are even more abstract) - like EKS, SQS, etc.

    The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.

    In my experience this often translates in values that flows to AWS, while the company giving value to customers is stuck with millions of cloud bills each month, and a large engineering footprint that eventually needs to cut, leaving fewer and fewer people working on the product.

    That said, I acknowledge that cloud has business reasons to exist, I wrote an entire other post about my hate for it, but I still acknowledge that. However there are some myths that finally are getting dispelled (outsource infra and focus on your product).

  • I mean, the person in question had "hardening EKS" on their CV. EKS still means that the whole data plane is your responsibility. How can you harden a cluster without understanding the foundation of container security (isolation primitives, capabilities, etc.)? Workload security is very much part of the job.

    I mean the moment some pod will need to run with some privilege (say, a log forwarder which gets host logs), and you need to "harden" the cluster, what do you do if you don't understand the concept of capabilities? I will tell you what, because I asked this very question, and the answer was "copy the logs elsewhere", which is the "make it work with the hammer solution" that again shows the damage of not understanding.

    I am with you about different scopes, skillsets etc. But here we were interviewing people with a completely matching skillset on paper.

    Oh yeah I see...

    As some old philosopher once said: "shit's fucked, yo".

    Seems to be appropriate here.

  • I agree with your lack of affection for cloud services, but I think your view might be a little skewed here. Does a senior mechanic need to understand the physics of piston design to be a great mechanic, or just gather years of experience fixing problems with the whole system that makes up the car?

    I'm a Senior Systems engineer. I know very little about kernel programming or OS design, but i know how the packages and applications work together and where problems might arise in how they interact. Software Engineers might not know how or don't want to spend time to set up the infrastructure to host their applications, so they rely on me to do it for them, or outsource my job to someone else's computer.

    But you know what the kernel is. You know that syscalls are a thing, you know what role the kernel performs, you know that different filesystems have different properties (and pros and cons), etc..

    You don't need to know the details, perhaps, but you can't ignore the fundamental theoretical concepts of kernel and OS. You might not know the whole detail of the boot procedure, but if your machines are stuck on boot, you know at least what to look for.

    Here I was talking about equally foundational topics. There is nothing "above" - say - producing attestations and then verifying them. That's literally all there is to it, but if you don't understand the theory behind it, what exactly are you doing? As as I said, I don't care about the details, I didn't expect someone mentioning ciphers or timestamp authorities, transparency logs etc. All I would expect is "we produce a signature with a bunch of metadata and we verify it where we consume the artifact, so that we are sure that the artifact has the properties attested by the signature".

    Not knowing this is like someone claiming that they administer Linux machines but can't explain what network interfaces are or how routing is determined. This is not a question of being expert on different layers, this is just being oblivious to those other layers completely.

  • A cloud VM, just shut it down and you're done.

    If this flexibility is needed, and it's an "if", a dedicated server does the same. But even a cloudVM is already lower level compared to other services (which are even more abstract) - like EKS, SQS, etc.

    The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.

    In my experience this often translates in values that flows to AWS, while the company giving value to customers is stuck with millions of cloud bills each month, and a large engineering footprint that eventually needs to cut, leaving fewer and fewer people working on the product.

    That said, I acknowledge that cloud has business reasons to exist, I wrote an entire other post about my hate for it, but I still acknowledge that. However there are some myths that finally are getting dispelled (outsource infra and focus on your product).

    I'd like to understand how self managing all the lower level components abstracted by the cloud is saving on headcount. Care to math that out for us?

  • Mind you that my take and experience is specifically in the context of security.

    I struggle to make the parallel that you suggest (which might work for some areas) with a security engineer.

    Say, a person learned to brainlessly parrot that pods need to have setting x or z. If they don't understand them, they can't offer meaningful insight in cases where that's not possibile (which might be specific), they can't provide a solid risk analysis etc.

    What is the counterpart to this gap?
    Because I struggle to see it. Breadth of areas where this superficial knowledge is available is useless, IMHO.

    Yeah I can see that.

    However, you are now arguing a different point than I am getting from your original post. Maybe my fault in interpretation ofc, but the main difference (in my view) is:

    You say "incompetent" and "less skilled" as general statements on senior engineers. Those statements are false.

    You also say "missing the skills you are looking for" which is obviously true.

    And the implication that before cloud, people developed the specific skills you need more naturally - because they had to. This makes sense and I believe it.

  • I'd like to understand how self managing all the lower level components abstracted by the cloud is saving on headcount. Care to math that out for us?

    It depends. An EKS cluster can cost easily 20x what an equivalent cluster costs with same resources. The amount of people necessary to manage it is very close compared to a bare cluster, which depending on the scale can save hundreds of thousands or millions per year, therefore allowing extra headcount.

    For example, a company I worked for had a team of 6 managing all their kubernetes cluster on rented dediservers. The infra costed around 50k/year. The same clusters on EKS could be managed by 4 people (maybe?), but would have costed easily 5-600k, especially since they were beefy machines, possibly even more. That amount of money would pay for 7-8 additional headcount in local hires.

    Considering that in those clusters there were 40-50 postgres clusters, if moving those to RDS they would have probably looked at millions in cloud bills per year, and the effort to run those dB's once the manifests were developed was negligible (same team was managing them).
    This was a tiny startup, with limited resources for internal tools and automation development.

    So it's not like managing everything can save headcount, it's that not outsourcing everything can save so much money that largely compensates for more headcount, plus you are giving money to real people, who spend local and pay taxes.

  • 48 Stimmen
    12 Beiträge
    2 Aufrufe
    E
    I can already speak to my cat. It's not really an enlightening conversation it's basically him demanding food, or he wants to go outside, or he wants better food than the food provided. That's basically the extent of his conversation skills. I mean he's a cat, he's not exactly going to talk about politics with me is it even if we could translate between our "languages"
  • 262 Stimmen
    49 Beiträge
    3 Aufrufe
    D
    They are examples of complex and difficult tasks that humans are capable of when working together, implying through comparison reordering society is also achievable.
  • 424 Stimmen
    80 Beiträge
    353 Aufrufe
    S
    It really depends on the company. Some look for any way to squeeze you. Others are pretty decent and probably more efficient as they dont waste as many working hours on bullshit claims and claim resolution. Also if i rent a car i want things to go smoothly. I got places to be. You make my life easy, ill happily pay again and do my best to make yours easy too.
  • AI Pressure from the Top: CEOs Urge Workers to Adapt

    Technology technology
    1
    1
    1 Stimmen
    1 Beiträge
    11 Aufrufe
    Niemand hat geantwortet
  • No JS, No CSS, No HTML: online "clubs" celebrate plainer websites

    Technology technology
    205
    2
    772 Stimmen
    205 Beiträge
    671 Aufrufe
    R
    Gemini is just a web replacement protocol. With basic things we remember from olden days Web, but with everything non-essential removed, for a client to be doable in a couple of days. I have my own Gemini viewer, LOL. This for me seems a completely different application from torrents. I was dreaming for a thing similar to torrent trackers for aggregating storage and computation and indexing and search, with search and aggregation and other services' responses being structured and standardized, and cryptographic identities, and some kind of market services to sell and buy storage and computation in unified and pooled, but transparent way (scripted by buyer\seller), similar to MMORPG markets, with the representation (what is a siloed service in modern web) being on the client native application, and those services allowing to build any kind of client-server huge system on them, that being global. But that's more of a global Facebook\Usenet\whatever, a killer of platforms. Their infrastructure is internal, while their representation is public on the Internet. I want to make infrastructure public on the Internet, and representation client-side, sharing it for many kinds of applications. Adding another layer to the OSI model, so to say, between transport and application layer. For this application: I think you could have some kind of Kademlia-based p2p with groups voluntarily joined (involving very huge groups) where nodes store replicas of partitions of group common data based on their pseudo-random identifiers and/or some kind of ring built from those identifiers, to balance storage and resilience. If a group has a creator, then you can have replication factor propagated signed by them, and membership too signed by them. But if having a creator (even with cryptographically delegated decisions) and propagating changes by them is not ok, then maybe just using whole data hash, or it's bittorrent-like info tree hash, as namespace with peers freely joining it can do. Then it may be better to partition not by parts of the whole piece, but by info tree? I guess making it exactly bittorrent-like is not a good idea, rather some kind of block tree, like for a filesystem, and a separate piece of information to lookup which file is in which blocks. If we are doing directory structure. Then, with freely joining it, there's no need in any owners or replication factors, I guess just pseudorandom distribution of hashes will do, and each node storing first partitions closest to its hash. Now thinking about it, such a system would be not that different from bittorrent and can even be interoperable with it. There's the issue of updates, yes, hence I've started with groups having hierarchy of creators, who can make or accept those updates. Having that and the ability to gradually store one group's data to another group, it should be possible to do forks of a certain state. But that line of thought makes reusing bittorrent only possible for part of the system. The whole database is guaranteed to be more than a normal HDD (1 TB? I dunno). Absolutely guaranteed, no doubt at all. 1 TB (for example) would be someone's collection of favorite stuff, and not too rich one.
  • Founder of 23andMe buys back company out of bankruptcy auction

    Technology technology
    60
    1
    348 Stimmen
    60 Beiträge
    249 Aufrufe
    A
    Come on up to Canada, we still got that garlic bomb. I can still taste the one from last week
  • 275 Stimmen
    68 Beiträge
    260 Aufrufe
    A
    intellectual property is grotesque. under no circumstances a creator should be barred from his creation. if shit like that happens I'd rather there not be any intellectual property at all
  • 258 Stimmen
    46 Beiträge
    196 Aufrufe
    stzyxh@feddit.orgS
    yea i also were there at a few thousand I think and the content has changed a lot since then.