CISA released an urgent message warning water utilities to disconnect their logic controllers from the internet in the face of rising cyberattacks.
The hacks target internet-facing programmable logic controllers (PLCs) that control equipment and allow machinery to communicate. They monitor and control the water pressure, chemical dosing, and other factors to ensure the water is safe.
Many of the PLCs are apparently open to the internet and use default credentials, allowing a remote attacker to easily take them over.
I assume it is this CISA: https://en.wikipedia.org/wiki/Cybersecurity_and_Infrastructure_Security_Agency
I will take things that never should be connected to Internet for $200 Alex.
Exactly! Whenever I see one of these stories of a utility being hacked that’s my first thought.
Not saying I ever accessed it, but there’s a generator in Germany open to the web without security. It makes a few MW. My guess is hospital or any other critical infrastructure.
Went on the public tour of our local waste-treatment facility. Really fascinating stuff. Highly recommend it if you’re even slightly curious.
There are definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.
But there were also office equipment and personal devices everywhere. All it takes is for someone wanting to monitor the state of a digester on their laptop from home.
You just crossed the beams.
Automation engineer here who does a lot of SCADA
There are soooo many good ways to securely monitor stuff from home. But especially in critical infrastructure it’s just usually air gapped which is unbeatable in cyber security.
Tours of any local facility are super cool! I find it all so interesting, seeing how the things we use work!
I worked on new SCADA for a water company across all theirs sites. Previously they were a mismatch of different systems and all airgapped.
They standardised everything and put them all on VPN. Suddenly you could access the secure water treatment works with 24/7 security guards and the control room was a bunker, from the unmanned rural sewage works with a portacabin for the server, the key hidden under a brick, and wifi for the PLCs.
The sewage works had always operated like that because it was deemed low risk. No one considered the VPN changed the threat model.
not so funny when it isn’t happening in the middle east but right at your doorstep
This has been talked about for years and about how exposed SCADA is. I just assumed that it was fixed/isolated years ago. The fix appears to have been to stop talking about it.
That’s why NIS2 in Europe demands that OT is on a separate network from IT. My old workplace (well, the head of FO Technology and his gang anyway) was up in arms about it because it “was a waste of time” (and completely ruin the ideal architecture he had in his mind).
The hacks target internet-facing programmable logic controllers (PLCs)
Why the fuck
is your PLC
facing the fucking internet!Jesus Zombie Christ an a pogo stick. Has no one been paying attention for the last two decades? Seriously, we learned this sort of lesson in Two Thousand and fucking Three. Your critical assets do not get public IP addresses.
It usually happens in 2 steps.
First step is that everything gets connected to the LAN and you can only access the PLC network from within the building’s network. Then some time later, management finds out that keeping someone on-call to go out costs a fortune, so they request that access be made so they can make the change from anywhere.
The IT team argues security, but no new hardware can be provisioned and a developing a new process is too hard. Then the magical phrase is uttered: “Just make it work”. So IT punches a hole in the firewall, adds a NAT rule, and job done.
Literally this. It starts with “everything needs to be on an air gapped LAN so we can control it.” And then suddenly that air gapped LAN isn’t air gapped, because someone got tired of paying the call-out fee every time the on-call person had to drive to the site to push a few buttons on a keyboard. But figuring out a proper VPN (or at least a reverse proxy with access control) was too difficult or expensive, (management thinks the point is to save money on call out fees, not spend money on proper hardware) so they just poked a bunch of port-forwarded holes in the firewall and called it a day.
What’s extra ridiculous is VPNs are not that hard. I am fantastically awful with networking and even I can wrap my head around a WireGuard tunnel.
The issue is doing one at scale, with proper access controls. Also, let’s not skip past the whole “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene, etc… All without any additional IT support or hardware, because management is doing this as a cost cutting measure.
I can pretty much guarantee that the systems that get hacked aren’t the ones that do everything right. They’re the ones who have a single shared VPN password, haven’t updated their firewall since it was installed 8 years ago, and they haven’t even changed the default passwords for their control systems. Because they got tired of answering phone calls about what the password was, so they just made every username and password something stupid like admin/admin.
Even when a company is doing things pretty well with a VPN, all it takes is one mistake and the cat is out of the bag. At one of my previous employers, we had a fairly good setup with VPNs using single sign on via our IdM provider. IdM enforced SSO and the VPN had a number of policies setup which kept good control over what users could login. And then someone had the brilliant idea to take an old VPN appliance, reset it to factory defaults and configure it just enough to work on our network and then hung it out on the internet for some sort of test (our configuration management left a lot to be desired). Of course, that led to it being left there for a couple months and an attacker finally found it. They also had a valid username/password combination which got them in the door.
Security is hard and often expensive. But still, compared to “we put out OT devices on the internet” a basic VPN is a huge step up.
Sadly yes, this is exactly what happens. And it ends up being IT holding the bag at the end, with the managers having long since cashed out their stock options and left.
“Well, we just bought this new monitoring platform from my cousin’s company and they need access to it. No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. Ok, then just open all ports to the network so they can connect. No, they don’t know what port or protocol, it also runs out of my cousin’s house so it’s on a dynamic IP so you can’t create some type of access rule for it. Just put an any/any rule in and it’s good. What do you mean we already have a monitoring platform? Well, this one is better!”
“No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. […]”
They won’t know that much. They don’t understand that their job isn’t running a water department, it’s running the computers that maintain the water department and that includes security. For someone familiar with computer security it’s easy to recognize negligence, but until the law sees it as criminal negligence nothing’s going to change.
Small town municipal water departments aren’t going to be able to afford the guy holding the shield against international state actors.
I did some work for a company that owned a /16 public IP range and used it on their internal network. I had to put a safety rated control system on it that ran mining machines that weighed 60 tons and cost a couple of million bucks each.
They gave me a subnet range for the system that was clearly non-private and I was like, “ummmmm, is that on the Internet?”
Apparently it wasn’t routed to the rest of the Internet but it always felt like it was one misconfigured router away from disaster.
2003??? We were learning this shit from Ferris Bueller back in the early 80s!!!
Why in the everloving fuck would scada ever be connected directly to even the regular lan, let alone the Internet!?!
Laziness. Poor training. Stupidity.
Why wouldn’t you want a key utility controlled from around the global?
why is any critical control equipment internet accessible.
I understand some machines might need internet to send reports and make document filings and shit. But it should not be the critical control equipment that makes the system function. And there should be a massive physical gap between the internet machines and the critical machines.
Jesus christ, even by the abysmally low bar I have for humanity, it still finds ways to dig itself a path under it.
The entire tech sector screamed about the vulnerability of IoT devices, plenty of which are now participants in zombie botnets.
Does anyone remember in 2015 when YESCO billboards by the thousands were hacked to show Goatse instead of an AT&T ad? Pepperidge Farm remembers. IoTs are still vulnerable, including big municipal ones and, yes, Flock Safety cameras.
And CISA
was severely brokenhad its staff severely reduced by DOGE.In unrelated news the CDC, also a victim of DOGE, is entirely unprepared for the next epidemic.
I am repeating what I said elsewhere on a different comm:
I call FULL bullshit on this being Iran.
I personally worked on the cybersecurity incident that happened to St Paul municipal and water systems last year as the Incident Response Technical lead.
The threat actors were not Iranian then. They were Russian.
I personally rebuilt most of their networks, VMware hosts, storage, and servers. The majority of the VMware environments did not have config backups and were running 5 and 6 branches of VMware, only two had 7+.
Their network was insecure. They had IPSEC VPNs between the various location, but there was no further security in that respect. Their firewalls were not being kept up. AAAAND… I warned them that we thought they were still compromised.
In fact, the forensics team called them out as being clean on a group call one week into the incident, but while we were on the call with leadership, they got RE-infected.
I had my team compile a massive set of recommendations for them and I wrote the report. Water Systems specifically. They did not listen to us, apparently. The forensics team also did this.
They had FBI, national guard (fucking incompetent), and law enforcement (useless) in all meetings.
If they got into Water, they’ll get back into city and municipal again. I guarantee it.
This is not Iran. It’s the fucking incompetence of city government and Russian threat actors.
Minor nit: Russia and Iran are already working together on shaheds and targeting
No surprise at all if they also coordinate cyber
Whatever. I worked the fucking case. The infiltrators were Russian. After that, it’s a political or financial negotiation.
If budget weren’t an issue, how many clones of your teams would it take to do that for all the water systems in the US in five years?
What are the odds that the average muncipality understands the security implications of just existing on the internet these days?
No doubt so trump can make up some bullshit excuse to try to suspend the midterms.
We already know he’s going to try, I hate how obvious this shit is from a thousand miles away, but still nothing is done about it.
A lot of this is often down to shitty IT practices by the vendor who has little IT knowledge let alone decent security insight.
I helped support a new PCL installation (technician control PCs and software upgrade) in this exact scenario many years ago. The vendor came in expecting to treat our system as their own. Direct remote access to several over-specced, physical servers from the outside world, available on our core network, shared account names (and passwords!) across all devices, every account a local admin and their own domain admin account so they can make changes as required… Needless to say our security guy almost broke a rib from laughing so hard!
What they got was very different. I ended up doing fairly basic troubleshooting to get the shitty software working without it needing admin rights to run - XP era software running on Win7 clients so it was meeting basic folder and reg key write permissions - everything else was locked down as tight as we could. They didn’t even get admin on the servers.
A few years later I heard they’d reported that our site was their most reliable. Probably due to the fact that they (or the techs) couldn’t fuck with anything without our permission. Apparently their original configuration is normal in the industry… Shudder
Yup, Sleepy-Don really do have the whole Iran situation under control.
it actually depends on it to keep distracting the msms, from talking epstein.
According to our peerless leader, I blame it on Minnesota because they are grossly incompetent
But then he tells California (and Canada) we are to blame for wildfires for our failure to rake our forests.
So are we just kind of admitting that there exists no way to expose any networked device to the internet securely? Because if it’s not possible for PLCs I don’t see why it would be possible for any device. If water utilities have to take these offline, then how does that advice not apply for every internet-capable device in every commercial and industrial facility worldwide?
There are ways, but they are costly and complicated and do not stop all attacks. So the golden rule is that if critical infrastructure doesn’t need to connect to the internet, it shouldn’t.
So are we just kind of admitting that there exists no way to expose any networked device to the internet securely?
It’s done all the time. It’s not a product you can buy, though, it’s a process. There’s probably a good “go by” for how to start the process for citywide infrastructure, but it’s just one more document for people to ignore.
I don’t work directly with PLCs but we do have them at my work. The main thing that makes these different from any other devices is that they typically control physical machines. Which means there is a real danger that them becoming compromised could lead to damaged equipment or even death.
Additionally, many PLCs use older OS versions like xp because their software is notoriously out of date (if it’s working why rewrite it when that could introduce safety concerns)
This “urgent” message from CISA does not prove that any of these devices are internet accessible just that if they are then they should be removed. If CISA wanted to require this then they would have released a binding operational directive (BOD) or an emergency directive (ED). Both of which are publicly viewable on their site.
disconnect […] from the internet
And these guys really think that this is doable as easy as snapping your fingers?
hmm…
And these guys really are national advisors on things related to computers and internet??
hmmmmm…
The specific advice they give is to disconnect all PLCs and only access them through a VPN, enable password protection and change the default passwords, and only allow IPs from known engineering laptops and other critical assets.
This is absolutely doable and any IT professional could manage this change. The only barrier that prevented this from happening before is that you need buy-in from someone with access to the purse strings
So what do you think they should do? Just ignore it?
