St. Paul, MN, was hacked so badly that the National Guard has been deployed
-
I'm sorry, Dave, but AUR does not count.
Precisely. The AUR is just a somewhat organized script dump. There's no release process, and any user can upload any script they want. If you're not capable of auditing scripts yourself, don't use the AUR, there's no expectation of quality or safety at all.
-
Exactly.
When there's a high profile bug in an important FOSS project, everyone and their dog is looking for a fix. Usually it'll be patched within days, if not hours, of being reported.
When there's a high profile bug in a closed size source project, the company backing it will deflect and delay until they're forced to fix it, and they can sometimes get away with it for years or even decades.
All software has bugs, which remain strategy do you prefer?
I mean, myself personally, I prefer to simp and fanboy for my favorite exploitative corperate overlord, because I'm sure there are good reasons everyone uses them, despite their well documented history of massive fuckups and fuckovers of all possible kinds!
/s
-
FBI, National Guard assist St. Paul as cyber-attackers force shutdown of Internet-based systems
St. Paul Mayor Melvin Carter declared a state of local emergency on July 29 following a days-long cyber attack on the city's Internet-based computer networks that led the city to call in the FBI and Gov. Tim Walz to enlist ...
(techxplore.com)
Minnesota activates National Guard as cyberattack on Saint Paul disrupts public services | TechCrunch
Gov. Tim Walz activated the state military's cyber forces to help ensure public services continue to run as the city of Saint Paul battles an ongoing cyberattack.
TechCrunch (techcrunch.com)
So, this actually was first detected on Friday July 25, escalated all the way up to the Emergency Operations Center on July 28 (Monday), state of emergency / near total intranet shut down (they are quarantineing the whole system) on July 29 (Tuesday).
It seems to me that some kind of rather sophisticated threat actor managed to get into the core ... this techxplore article calls it a 'VPN', but it isn't technically a VPN, its a secure access tunnel system that city-gov systems and employees use to talk to each other, it almost certainly is not intended to be geared toward broad internet access/usage, beyond accepting user input from public facing government web portals, such as say, people paying their utliity bills online or trying to submit a business liscense application online, things like that.
This system is sounding like it got fully compromised (as in, low level/high privilege level access was secured), and was either sending data out/in through improper IP addresses, and/or was possibly being hijacked to do some kind of DOS attack ... on itself?
I am having a really hard time finding any exact details on this, but this is my best guess.
Given that the EOC essentially immediately shutdown everything and called in a National Guard Cybersecurity team, it seems to me that there is a high chance this was done by basically a nation-state level threat actor.
It also at least seems like the systems, the data, the hardware, have at least not yet been locked down in a ransomware style move, which... could be largely due to their just quickly pulling the whole thing offline, or could be because that wasn't the goal of the attackers... or some combination of both.
but it isn’t technically a VPN
It is. Others have given some details, but I'll keep it simple.
A VPN makes remote devices seem like they're on the same network. You can have all traffic be routed through that virtual network, or just some of it. Common use cases:
- consumers - make yourself appear to be somewhere else; basically replaces old SOCKS proxies (all traffic routed)
- workplace - provide access to internal, protected resources to those that need them (only relevant traffic is routed)
- home lab - expose internal services publicly (reverse of workplace use case)
Those are all VPNs, though the first is acting more like a proxy than the others.
National Guard Cybersecurity team
This isn't some crack team of experts, it's mostly part-time soldiers who likely have a relevant day job. My brother-in-law is a mechanic at the National Guard, not because he's an expert, but because they paid for his 4-year degree and only expect a few hours of work each month. A lot of people join for inexpensive medical insurance.
This cybersecurity team is probably just a handful of locals who work in IT locally and have had training on systems commonly used by the military.
If this was a high profile attack by a state actor or something, they wouldn't call the National Guard, they'd call the NSA, CIA, or something similar, as in an actual crack team. The National Guard is mostly there to provide structure in emergencies, like organizing rescue efforts in a flood or help firefighters with labor in fighting wildfires. They're just weekend warriors, not experts.
-
Or some bored teenager somewhere
That sounds much more likely. I don't care about St. Paul and I'm American, why would China or Russia care? Also, state and city governments all handle things differently, so the only takeaway is that St Paul's IT is probably incompetent.
-
The article says it started on a Friday morning in Minnesota. It’s clear that that’s when the attack started and not a case of the first guy starting work that day discovering that it happened, because the article also says that they tried to contain it as it was going on, but ultimately failed.
Minnesota is at UTC-5 and China is at UTC+8, meaning when it’s morning in Minnesota, it’s already 13 hours later in China, i.e. middle of the night.
It's probably a local.
-
but it isn’t technically a VPN
It is. Others have given some details, but I'll keep it simple.
A VPN makes remote devices seem like they're on the same network. You can have all traffic be routed through that virtual network, or just some of it. Common use cases:
- consumers - make yourself appear to be somewhere else; basically replaces old SOCKS proxies (all traffic routed)
- workplace - provide access to internal, protected resources to those that need them (only relevant traffic is routed)
- home lab - expose internal services publicly (reverse of workplace use case)
Those are all VPNs, though the first is acting more like a proxy than the others.
National Guard Cybersecurity team
This isn't some crack team of experts, it's mostly part-time soldiers who likely have a relevant day job. My brother-in-law is a mechanic at the National Guard, not because he's an expert, but because they paid for his 4-year degree and only expect a few hours of work each month. A lot of people join for inexpensive medical insurance.
This cybersecurity team is probably just a handful of locals who work in IT locally and have had training on systems commonly used by the military.
If this was a high profile attack by a state actor or something, they wouldn't call the National Guard, they'd call the NSA, CIA, or something similar, as in an actual crack team. The National Guard is mostly there to provide structure in emergencies, like organizing rescue efforts in a flood or help firefighters with labor in fighting wildfires. They're just weekend warriors, not experts.
I guess my confusion here comes from trying to reconcile the broad, colloquial understanding of a VPN, and the actual, precise, technical definition.
When a news article runs with VPN in a wide audience usage... 95% of people think SurfShark or Nord or PIA or whatever, something that is consumer oriented, that accesses/fancy proxies the broad internet, as you give in your first example, where it basically functions as a more elaborate set of proxies than what most people could probably manage on their own.
So... yes, it technically is a type 2 VPN as you've listed, but it technically isn't a type 1 VPN, which is what 95% of people think a VPN is.
I've worked remote for a decently long while, and most other remote workers I've known... they do not have really any understanding at all that their work login thing... is fundamentally the same kind of VPN as Surfshark, just configured differently.
My goal was to emphasize this difference, but yeah, I could have used better wording.
And yes, I know as well that Nat Guard CyberSec are by no means the creme de la creme of cybersec specialists, but the fact that a top level Municipal agency went 'oh fuck' and basically escalated the issue to the next level of IT support, the State Nat. Guard... that means they got pretty fucking spooked.
Also, the FBI is involved as well, they'd be the ones to pass it up to NSA and/or Homeland Security, I think... and the Nat Guard would be the ones capable of passing it up to... Army CyberCom... and I think if it makes it up to either Army CyberCom or the NSA or Homeland Sec, well at that point, its theoretically possible that any member of the alphabet soup could be called upon, or at the very least, have it come up on someone's desk.
I am not exactly sure what the CoC of escalation pathways is here, but it seems like this got escalated to as many people as the Municipal Emergency Response Team could, quite rapidly.
Its 'the emergency response team looked at this for 24 hours and then called in another emergency response team'.
-
I guess my confusion here comes from trying to reconcile the broad, colloquial understanding of a VPN, and the actual, precise, technical definition.
When a news article runs with VPN in a wide audience usage... 95% of people think SurfShark or Nord or PIA or whatever, something that is consumer oriented, that accesses/fancy proxies the broad internet, as you give in your first example, where it basically functions as a more elaborate set of proxies than what most people could probably manage on their own.
So... yes, it technically is a type 2 VPN as you've listed, but it technically isn't a type 1 VPN, which is what 95% of people think a VPN is.
I've worked remote for a decently long while, and most other remote workers I've known... they do not have really any understanding at all that their work login thing... is fundamentally the same kind of VPN as Surfshark, just configured differently.
My goal was to emphasize this difference, but yeah, I could have used better wording.
And yes, I know as well that Nat Guard CyberSec are by no means the creme de la creme of cybersec specialists, but the fact that a top level Municipal agency went 'oh fuck' and basically escalated the issue to the next level of IT support, the State Nat. Guard... that means they got pretty fucking spooked.
Also, the FBI is involved as well, they'd be the ones to pass it up to NSA and/or Homeland Security, I think... and the Nat Guard would be the ones capable of passing it up to... Army CyberCom... and I think if it makes it up to either Army CyberCom or the NSA or Homeland Sec, well at that point, its theoretically possible that any member of the alphabet soup could be called upon, or at the very least, have it come up on someone's desk.
I am not exactly sure what the CoC of escalation pathways is here, but it seems like this got escalated to as many people as the Municipal Emergency Response Team could, quite rapidly.
Its 'the emergency response team looked at this for 24 hours and then called in another emergency response team'.
So… yes, it technically is a type 2 VPN as you’ve listed, but it technically isn’t a type 1 VPN, which is what 95% of people think a VPN is.
Sure. But VPNs were around long before the consumer-oriented VPNs were a thing.
spooked
Or they just had one person handling their IT and needed help, and didn't want to pay an outside contractor.
I'm honestly surprised the National Guard was called at all. If anything, that shows how backwards Minnesota is, or at least the mayor of St. Paul. I'd expect that if my state government got hacked, they'd call in a local cyber security firm to come audit things, and we have plenty of them here (I'm in Utah, so not even a big state). This isn't a National Guard situation, it's an independent cyber security audit and FBI situation.
Here's how I expect this happened:
- St. Paul's small IT team escalated the issue to the mayor because they were overwhelmed
- Minnesota Governor (Tim Walz) didn't know what to do, so he called everyone, including the National Guard
- everyone responded
-
So… yes, it technically is a type 2 VPN as you’ve listed, but it technically isn’t a type 1 VPN, which is what 95% of people think a VPN is.
Sure. But VPNs were around long before the consumer-oriented VPNs were a thing.
spooked
Or they just had one person handling their IT and needed help, and didn't want to pay an outside contractor.
I'm honestly surprised the National Guard was called at all. If anything, that shows how backwards Minnesota is, or at least the mayor of St. Paul. I'd expect that if my state government got hacked, they'd call in a local cyber security firm to come audit things, and we have plenty of them here (I'm in Utah, so not even a big state). This isn't a National Guard situation, it's an independent cyber security audit and FBI situation.
Here's how I expect this happened:
- St. Paul's small IT team escalated the issue to the mayor because they were overwhelmed
- Minnesota Governor (Tim Walz) didn't know what to do, so he called everyone, including the National Guard
- everyone responded
Sure. But VPNs were around long before the consumer-oriented VPNs were a thing.
No argument there, you're right.
(technically =P)
Or they just had one person handling their IT and needed help, and didn't want to pay an outside contractor.
Nah, read the links I provided.
It went from the normal IT department, to the city level Emergency Response Team, to the Nat Guard and FBI.
Cities, larger ones anyway ... often have their own sort of local mini-FEMA, who have their own capacities to order around other local agencies, but also have a whole bunch of protocols for... who to contact when something exceeds the capacity of everything they can more or less order around with their own authority.
I'm honestly surprised the National Guard was called at all. If If anything, that shows how backwards Minnesota is, or at least the mayor of St. Paul.
I am not in particular familiar with St.Paul specifically... but ...
-
It could overall make sense given the capacities of the city (the Twin Cities, St. Paul + Minneapolis), and them knowing their own constraints.
-
It could also make sense if they rather rapidly at least suspected a very sophisticated, foreign threat actor.
That second half is kinda most of my argument:
Why would you start up the Military chain of escalation unless you either suspected a potential foreign nation state actor, and/or, critical infrastructure systems were breached, so critical that they'd been previously deemed an actual national security risk, should that happen?
I am not certain of what happened, nor certain of the validity of this logic... but this is my logic, from the original comment.
Sure, they could have just panicked. I don't know that they did or did not.
But I have worked with people who've been employed by, led things like FEMA and DHS and City level emergency response teams, their specialities being the cybersec/netsec variety, and... this seems like actually following a previously outlined set of steps to me.
I'd expect that if my state government got hacked, they'd call in a local cyber security firm to come audit things, and we have plenty of them here (I'm in Utah, so not even a big state).
Ahahah, two things here:
-
Basically, see what I just wrote above.
-
Really? Utah, prime recruiting ground for the CIA, Utah, with the largest NSA data center complex in the country, possibly the world, that is archiving essentially all US internal communications they can so they can search through them later if need be, Utah, with more and more corporate datacenters all the time... you don't class Utah as a big state, in terms of the tech sector?
Perhaps I am misunderstanding you, but I just find that silly.
-
Sure. But VPNs were around long before the consumer-oriented VPNs were a thing.
No argument there, you're right.
(technically =P)
Or they just had one person handling their IT and needed help, and didn't want to pay an outside contractor.
Nah, read the links I provided.
It went from the normal IT department, to the city level Emergency Response Team, to the Nat Guard and FBI.
Cities, larger ones anyway ... often have their own sort of local mini-FEMA, who have their own capacities to order around other local agencies, but also have a whole bunch of protocols for... who to contact when something exceeds the capacity of everything they can more or less order around with their own authority.
I'm honestly surprised the National Guard was called at all. If If anything, that shows how backwards Minnesota is, or at least the mayor of St. Paul.
I am not in particular familiar with St.Paul specifically... but ...
-
It could overall make sense given the capacities of the city (the Twin Cities, St. Paul + Minneapolis), and them knowing their own constraints.
-
It could also make sense if they rather rapidly at least suspected a very sophisticated, foreign threat actor.
That second half is kinda most of my argument:
Why would you start up the Military chain of escalation unless you either suspected a potential foreign nation state actor, and/or, critical infrastructure systems were breached, so critical that they'd been previously deemed an actual national security risk, should that happen?
I am not certain of what happened, nor certain of the validity of this logic... but this is my logic, from the original comment.
Sure, they could have just panicked. I don't know that they did or did not.
But I have worked with people who've been employed by, led things like FEMA and DHS and City level emergency response teams, their specialities being the cybersec/netsec variety, and... this seems like actually following a previously outlined set of steps to me.
I'd expect that if my state government got hacked, they'd call in a local cyber security firm to come audit things, and we have plenty of them here (I'm in Utah, so not even a big state).
Ahahah, two things here:
-
Basically, see what I just wrote above.
-
Really? Utah, prime recruiting ground for the CIA, Utah, with the largest NSA data center complex in the country, possibly the world, that is archiving essentially all US internal communications they can so they can search through them later if need be, Utah, with more and more corporate datacenters all the time... you don't class Utah as a big state, in terms of the tech sector?
Perhaps I am misunderstanding you, but I just find that silly.
you don’t class Utah as a big state, in terms of the tech sector?
In terms of military, we have:
- one major Air Force base (Hill)
- some storage and testing facilities
- refueling facility for the Army
That's it. We have ~3.5M people (~1/100 of US population), and only ~3 metros that matter (SLC, Utah County, St. George). Minnesota has ~5.7M people, so it's almost twice as big, and the Twin Cities cities area (includes St. Paul) is bigger than the entire population of Utah.
So while Utah punches above its weight in tech, St. Paul area absolutely dwarfs it in population. Surely they have a robust cybersecurity industry there...
The National Guard just seems like a desperate move. When they're deployed, they take orders from the the federal military, and at peace, monitoring foreign threats seems like a federal thing. You call in the National Guard to put down a riot or something where you just need bodies, not for anything niche. The only way that makes sense is if they think there will be an invasion (angsty/Canadians?) and they need boots on the ground for physical protection. Otherwise, just call a local cybersecurity firm to trace the attack and assess damage.
-
-
you don’t class Utah as a big state, in terms of the tech sector?
In terms of military, we have:
- one major Air Force base (Hill)
- some storage and testing facilities
- refueling facility for the Army
That's it. We have ~3.5M people (~1/100 of US population), and only ~3 metros that matter (SLC, Utah County, St. George). Minnesota has ~5.7M people, so it's almost twice as big, and the Twin Cities cities area (includes St. Paul) is bigger than the entire population of Utah.
So while Utah punches above its weight in tech, St. Paul area absolutely dwarfs it in population. Surely they have a robust cybersecurity industry there...
The National Guard just seems like a desperate move. When they're deployed, they take orders from the the federal military, and at peace, monitoring foreign threats seems like a federal thing. You call in the National Guard to put down a riot or something where you just need bodies, not for anything niche. The only way that makes sense is if they think there will be an invasion (angsty/Canadians?) and they need boots on the ground for physical protection. Otherwise, just call a local cybersecurity firm to trace the attack and assess damage.
So while Utah punches above its weight in tech, St. Paul area absolutely dwarfs it in population. Surely they have a robust cybersecurity industry there...
https://lecbyo.files.cmp.optimizely.com/download/fa9be256b74111efa0ca8e42e80f1a8f?sfvrsn=a8aa5246_2
Utah, #1 projected tech sector growth in the next decade, of all 50 states.
Utah, #8 for tech sector % of entire state economy, of all 50 states.
Minnesota?
Doesn't crack top 10 for any metrics.
Utah may not be the biggest or techiest state, but it is way more so than Minnesota.
The National Guard just seems like a desperate move.
Again, this is my argument, but you are only seeing desperation as due to incompetence, not due to... actual severity.
When they're deployed, they take orders from the the federal military,
Not actually true unless the Nat Guard has been given a direct command by the Pentagon.
and at peace, monitoring foreign threats seems like a federal thing.
... which is why the FBI were called in, in addition to the Nat Guard being able to report up the military CoC.
You call in the National Guard to put down a riot or something where you just need bodies, not for anything niche.
I mean, you yourself have explained that the Nat Guard does have a CyberSec ability, and I've explained they also have the ability to potentially summon even greater CyberSec ability.
I guess you would be surprised how involved the military is / can be in defending against national security threatening, critical infrastructure comprimising kinds of domestic threats.
Remember Stuxnet?
Yeah other people can do that to us now, we kinda uncorked the genie bottle on that one.
Otherwise, just call a local cybersecurity firm to trace the attack and assess damage.
It is not everyone's instinct or best practice to immediately hire a contracted firm to do things that government agencies can, and have a responsibility to do.
If this was like, Amazon being comprimised, yeah I can see that being a more likely avenue, though if it was serious, they'd probably call in some or multiple forms of 'the Feds' as well.
But this was a breach/compromise of a municipal network... thats a government thing. Not a private sector thing.
EDIT:
Also, you are acting like either you are unaware of the following, or ... don't think its real?
Kind of a really big deal in terms of Utah and the tech sector and the Federal government and... things that were totally illegal before the PATRIOT Act.
Exabytes of storage.
Exabytes.
Utah literally is where the NSA is doing their damndest to make a hardcopy of literally all internet traffic and content.
Given how classified this facility is, I wouldn't be surprised if their employees don't exactly show up in standard Utah employment figures.